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ABSTRACT 



Apparatus and methods are disclosed for simultaneously 
tracking multiple finger and palm contacts as hands 
approach, touch, and slide across a proximity-sensiDg, 
compliant, and flexible multi-touch surface. The surface 
consists of compressible cushion, dielectric, electrode, and 
circuitry layers. A simple proximity transduction circuit is 
placed under each electrode to maximize signal-to-noise 
ratio and to reduce wiring complexity. Such distributed 
transduction circuitry is economical for large surfaces when 
implemented with thin-film transistor techniques. Scanning 
and signal ofifeet removal on an electrode array produces 
low-noise proximity images. Segmentation processing of 
each proximity image constructs a group of electrodes 
corresponding to each distinguishable contact and extracts 
shape, position and surface proximity features for each 
group. Groups in successive images which correspond to the 
same hand contact are linked by a persistent path tracker 
which also detects individual contact touchdown and liftoff. 
Combinatorial optimization modules associate each con- 
tact's path with a particular fingertip, thumb, or palm of 
either hand on the basis of biomechanical constraints and 
contact features. Classification of intuitive hand configura- 
tions and motions enables unprecedented integration of 
typing, resting, pointing, scrolling, 3D manipulation, and 
handwriting into a versatile, ergonomic computer input 
device. 

118 Claims, 45 Drawing Sheets 
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, lucu jau. ^, lipyo. j witbout the step of reposiUoaing the cursor over an 

BACKGROUND OF THE INVENTION on-screen controL Preferably the graphical manipulation 

A. Field of the Inveation channels should be distributed amoi^ many finger and 
The present invention relates generally to methods and motion combinatioiB to spread the woddoad. Touch- 
apparatus for data input, and, more particulaiiy. to a method ^ auxilliary scrolling controls such as the 
airi apparatus for integrating manual input ^° Cirque® Smartcat toucfapad with edge scrolling, the IBM® 

B. Description of the Related Art ScrollPoint™ mouse with embedded pointing stick, and the 

Many methods for manual input of data and comma«ls to et^'^H^ " " 1^^'^^^ '° ^'^'^ 

computeis ate in use today, but each is m^^VSTtn^ . TT"' improvements in this area, but stiU do 
easy to use for particular ^ of Sa iiJuTpoTe^iplf na e'Tlf f*!* manipulation channels to elimi- 
drawing tablets with peiK or oucks ,1 rS!' " ^1 ""'^ often-used cursor motion sequences, 
sketching, and quick co^J^^^^l^^^^ S^T"' " H ^f^- '""/'"^ ^ "'^^ 
a styh« is oom^nient for fiUinnuTforl wUdT^lS Z ^T^L!^ Proceedings of CHI '97 

signatures, special symbols, or LaU SZu^fte^^ S^of "-'Pulation of more than two 

handwriting is slow compai;d to typing and^« h ^ " * ^^"^ 

long doci^ents. Mice, 4er-sticb:SZ:^7x^l aI ^ --"I'aneous pamung. «K.ming and 

cursor pomting and graphical object manipulations such as A„„7h .u j r 

drag and drop. RoUeis, thumbwheels and trackballs excel at Another common method for reducmg excess motion and 
panning and scrollii^. Hie diversity of tasks that many automatically continue pointing or scrolling 

computer users encounter in a single day call for aU of these jpo^™*"' siffials once the user has stopped moving or lifts 
techniques, but few users wiU pay for a multitude of input " ^f^' methods can be distinguished by the 

devices, and the separate devices are often incompahble in ,Tn ^''^'^.^fSf """^"""ion ^ enabled, 

a usability and an ergonomic sense. For instance drawing - 1 . ' Z' ' ' ^'^^ continues image 

tablels are a must for graphics professionals, but kdtching ^^""^ T^" ^ l"^* ""^'^ °^ P°""i°e device 
between drawing and typing is inconvenient because the pen thresholds. Automatic panning is stopped 

must be put down or held awkwardly between the finwrs ^ ^L!?"'""^ P""""* ^^"'"^ ^ opposite 

while typing. Thus, there is a long-felt need in the art for a Slpppmg requires additional precise move- 

manual input device which is cllap yetTff^^Tenknt """f ^"'^ ^0-5,543,591 to Gillespie et aJ., motion 

integration of common manual imut tedmiques continuation occurs when the finger enters an edge border 

Speech recognition is an exciting. newtecLtogy which « "fj"^'?'** ^.^ '°"'='vaA Continued motion speed is 

promisestorelievesomeoftheSburden^ZS^ Sr^tSTtS^Trr ""^ 1"" ''^ 

However, voice is not appropriate for inpnttine all tvnes of ^ L ^ ^ edge. Cbntmuation 

data either. CurrenUy. v^ toput is be^SSoo Tt'"^ ^"J"^ '"^ "^ef"" °' ^ 

of long text documents. Un7natural la^S re«S^n P/fJ>^,'^-ly.'-» 
matures suflSdently that very high level\^ ^Sd^ T"^ 

canbeunderstood'bythe<S:nXvoL"'^harS " Z^^ts^^^Ttl^ZlZr''^^^^ 

advantage over keyboard hot-keys and mouse menus fhr "™?yi"g- ™- No. 5327.161 to Logan et al. describes 

command and coZl. Furthe^ref S »o^<« contmuaton when the finger enten, a border a^^ 

drawing, and manipulation of graphic^ S ii^St' ^nri™!" TT" «°"J^'io» ""ode. motion 

with voice commairis, no mattcT tow wefl iShfeZS ZT^I^^ ^ ^ " "^^^ °f fi°g« veloc- 

stood. Tims, theT^ always be a^T^e^t" Tt^L"''^"^- '^'T^''^'''^'' 

multi-function manual input devices which supotemenJ . ?J?^ °' "° ^ ^ » subsequent 

voice input "cvices wmcn supptement touchdown on the sur&ce. Disadvantageously, touch veloc- 

need to provide explicit mode switch sicnals such a* \m t. . 

buttonpresses, arm relocations, or styhs^^^f.^ 7l^T '^""^ P°'"'^ 

switching from one input activity to aiither E^tofcwT ^^^^ "^"'^ '° '^'^^ 

cal studies suggest^ repetS,^ ^rJ^"^,,^ ^JP-g PosiUoo to access the pointing device. Tliese inc^^^ 

causingrepeti.ives.raininSLAwkward^,t'vi^ " S^sSt^F^Z'^f ^ ."^ ''"^ 
activation force, wasted motion, and repeti^ sh)uW be a-,aJ^. T^I ^ ' ""^»t«l pourtmg stick 
minimized to improve eigonomi«TuS»re tte f^'?^** ^'l ^"i^ ^ "Force-to-Motion 

load should Tspread Evenly oveJ^iflS^ m^S^ '^TcX ^ISfiS^-'^T!^"'" 
groups to avoid repetitive strain. UVltKACT 90 pp. 701-06 (1990). and the position sensing 
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accessible from typing hand positions An ergonomic typing system should also adapt to indi- 

Touch screens and touchpads often distinguish pointine * Z^rt an.^f'^^T.If ^^^^'^"'^ ^ ^yle. and 

mouons from emulated button clicks or by ^^mlT^ L^u'^^.*'^ P"'*^ 

assuming very UtUe lateral fingertip motion^^^x^ ST ""=5*°^ P^^PO^ed, mechanical key- 

during tops on the touch surta« Xch ^ intSd^ ^teS^ '^.^ repositioned at great cost. For example, 

clicks. Inherent in these method^ ^mnh'^i,^ ^! "7^. """^"^ ^^^^^ '^^^ "y Har- 

topping will usually be st^S do^ fr^m^^ J^^ '° ^^''T ^-^^^^ bands well 

fiBger'pcsition. mLSiStci^S^ne^L^^.S W^Tt '^.^ ^ - « posiUon. A touch 

motion tangential to the surfaoelk^T^ a^S^^" , ''^f "'i:''* ''^^J' ^ '^'P^'^ or 

if the surface is not finely divided S>d^^,rZ.so, tf^^ r^^gm^ as long as the changes didn't con- 
if the user does a slow, "hunted J^SJ^L^f^ > ""^"^ ""^^ P'°^'= 

each key before striki^. ^rT^S^i^T^J"' '"'"f ^ tactile feedback as the edges of 

5.543.591 to Gille^t al. a Spal,^"^ J^tT^i forf^ ''^'cbes. nius. there exists a need in the art 

motions to the host computer aT cw^ mn^.Jf. f^ . «1 '^'J* ^cognizer which can adapt a key layout 

Howeve^ifthefingerisliftTsSnrnour^rrcE ^JL^^'X^i'^'^^'T^^^^T^'^ 
to count as a tap and if the accumulated btend mo^^^ dnfl due to hmtted toctile feedback. 

not excessive, any sent motions are undone a^ a .^^ ^ weU^t^h" ^^^^ » "^'yl-s 
button click is sent instead. This method only\^rrfo^ ^.h "^^ ?^ " '^^^^^ '^*' ^'^e«'« 
mouse commands such as pointing Xh S S t ^"'^J^'' '^""J"? '^""^ ^^'-^ """^ be put 
«ndone.notfordn.ggingoro^rm^puhtrLus P? Stif.'^r^'? °' held awkwardly during other i-^ut 
No.5.666.mtoLogan.topswithlessS^ab^my""kte2i !" ^ distinguish the hand- 
motion activate keys on a ^aU keypad wh^Tteralt^oS tT^u"^ "" .'•'"^ «>f « 
in excess of activates cuBor con^ nwS^^ to^h ^ a need in the art for a method to 
patents cursor mode is invoked by de&ult ^n a finder '^T .^'^f^^S g""*^ without a stylus and 
Stays on the surface a long time. ^ oonfusmg them with pointing motions. 

However, fast touch typing on a surface divided into , hJ^^^ ".l*! ""S" ""eds cited above could 

large array of key region^^top^oJ^tiSL ^'^'-g '«=b«'logy wbich di^ 

motions along the Irface than rela^^fcS a vanetyofhand configurations and motions such as sUding 

niques can tolerate. Such an array con^n^e w iTmul^: ^ k1 ""^ ^^^^ "'ecbanical chord keyboart^ 

^ and cohmms which may 'ooTt ^yl^.te ^yl^'^T^'^''''^'!!^'^'^^^^^'^'^^ 

fingers, so the user must readi with theS^ ^fex « ^ ^^^^'^'^'^''^^'^'^"^^''^^''^''^^'^^ 

extend fingers to touch many of the keyTdo^ ^icl^ . V * "^S"" ^"^^ «vSal 

n=aching and extending impL^ s£il^t C S^hvT''^'"'"'"'?''^""^'''^'""^'''"-"— 

motion while the finger is in the drwhich ma/s^ S^TL^ ) ^"t^ '^^^ 

present when the finger contacts the surface GtociSteS a^.^ capacitance of or between elongated wires which 

with as much as %' lateral motion measuS alh^^^ ^ ^ "'l'™^^- A thin dielectric is 

can easfly result. Attempting to Tte"^™,! ^T^l ^^y*'^ P'^ence of 

motion would make the ™ seS^ e£^ '^'i'*: '""'^ »«"^y 

sive. Furthermore, it may be desirabteTwteTa ^Sc m^f^" °^ technologies use projective 

or automatic key repeat' mode instead of StotSTS 'f.'T' ^'^^ ^"^^ °° ""^ '^^^ 

when the finger is heU in one place on tf,r^i^ T ^ ^ ^ °^ ^ » P'^l^ °' "'I"'"", 

lateral shifting by the fingertip SS^ a J^l^fi^^ ^^y "-"'-•q-ly deten^^ 

press would also be picked up as cms^ jitt^rSa S , Multipl^Touch- 

filtering. -nms. there"^ is a nL i^ i^lor a meil^^ S^l'n q^"L'^k!"*L '"^''^"^ °' '^'^^ 

distinguish keying from pointing on thVsa^e s^a^vS P-^^/ ^^^^T^b^! 'bey can do is «x,unt fingertips which 

more fobust hand configuratiorToies than laS^ral mo1^„Tf c • '^'^ 'bat will fail if a 

a single finger. ^ ^ ""'''^ '° ^-^b or palm .s introduced in the same column as a 

A . . fingertip. 

An eigononuctypmg system should require minimal key In US Pal Nn^ 1 </;«: ^ c,ncA„ ^ 

tappmg force, eaaly distinguish finger tans finn, rP^.„„ i u 5.565.658 and 535.017. Geipheide et 

hands, and cushion'the fin^TfiZLl^^fo'^f J^^^ tbe -utualcapacitan* between row and column 

surfiice impact Mechanical and membrane S^STreW « by dnvmg one set of electrodes at some clock 

onthespdngforceinthekeyswitebS tL^^J^aSn JZIT ! ^^^^^^^y » 

when the hands are resting on the keys^i^ " «'• S"** synchronous 

irreconcilable tradeofi^ between the e^onoSc to ^^^^ •» noise at the driving 

reduce the fatigue from key activating fo^i^dto^^.he ^ 1° ""^"^ signal-to-noise ratio they fo™ 

full weight oflhe hands oLIK^s^ r^^rfo^ 60 12? T'"^^ '""''^'^ '"'^ '°"l'iP«« 

Force minimization on touch surf^^sT^p^bte ^ tZJ^' "u"^^' "''^ «^ 

capadtive or active optical sensing,^ch d^t relJ^ ^ ^ ^^"^ oombmations until the various mutual 

fingprpressure. ra.he^'than resistivt^tt ^ ^ °" ^'S^^th'^e orodL" f'.t!^''- T''^ 

acoustic-wave sensing techniques TTie related art t^^h ^TT? P""^"** °^ ""^ '^^ cohimns in each 

devices discussed betow wiuTc^me «i^if^ £J « a ^t ""'•^ ^'"'^ 

hand, inchiding its four fingertips, a thi^bwd • ^ signaJ-to-noise ratio for virtual elec- 

pahn heels, res^ on the suZ.. T^lherexiS a Sit K^vf "^"^"^ '?'»"'-« «'ws and two columns. 

mere exists a long felt However, to uniquely distinguish multiple objects, virtual 
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than a finger tip. i.e. aboutn^ ^^^r^^l^^^'L T T *° ^ i-'^'Polates only on 

will degrade L sien^l-toZ^^^a^^'^ f '^e eight neanst neighbor electrodes ofeach local maximum 

noise ra'S,dn,psT^w'Td'"coC^;,^;St 5 '^'Zt^^'^'^'^'''"^^^'^^^^^^^ 
cover a large area ^ to 5 Mrf pahn contacts may cover more than nine electiodes. 

In U S PaL No^ «: -iA-^ «:oi c ^^-^ co« . . a need in the art for improved means to 

rZ^t .1 5-5^3.591. 5.543.590, and 5.495.0T7. group exactly those electrodes which are coveted by each 

?o?^^l r^^^:^'^'-^'^"''''-^'^'^^ di-'i-euishablehandcontactandtocomputeTSt^dtom 

forrow and cohimn electrodes independenUy. Total elec- such potentially irregular groups a«niroidtiom 

trode capacitance is estimated by mcasurine the eledmde lo . 6" f • 

voluge Changs caused bv intectina orl^Lv! !^ ^ maximum advantage of multi-touch surface 

measured simultaneously tfe^ elertr«i^^-,! i"*''' ""^ P*^ °^ contacting the 

driveWcircuit.TT.Sni.iScStm^L'Z S« I""- ""'^ '^^^ ^^1^ P^^^ "^'^ 
column electrode signals establishes an inte^hted v^^I 15 ™''S«:P'»"™'y ™jiges provide clear indications of where 

and horizontal posid^nfoHl^e^bTecT SeiX^ T*^"*^ ""dultered by himinosity 

in general havrWgher siirtfnoSn^rJ^h.^'^ °? ^^o^^ objects in the background. TT..^ 

nous methods, but Ae signTtol^.afe I S £Z ""'T ""."^ segmentaUon stages can be 

asrowandcoiumnlengltein^ri£S.tti^^u^^ Z^L^f r"""" ""t"!. """P"'" ^*''>'' 

especiaUy important for accurately locS oS wUch 20 -TSe Re .^"^tn u'^^'^^^ ^- 

are Ooating a few millimeteis above the p!d tKT^ ^^^11^^^^^'^^"'^'^"^^''^"^ 

method can detect such objects, it tend^ to ^^TtiS fonjLtZn 2 vSTfE^E nA'^T' ".1 

posiUon as being near the middle of the pad or sirnnlv dne*: w.Z^w \i e . (^^^) Cm and J. 

not detect floatkg objects near Ite S ^ ^ Hand Sega,entaUon Using Learning-Based ftedic 

TK,,e *K • * . ^ Verification for Hand Sign Recoenition " Proceed 
Thus there exists a need in the art for a capacitance- ^ ings of the 1996 IEEE Camper S^ Co^er^Tt 

sensing apparatus which does not suffer from pJor signal- Compter Vision and P^m R^^H^ f^ 

to-noise ratio and the multiple finger indistin^h;l5ity (199^. However pa^ ofVh^7Th^^^^ 

pn>blems of touchpads with long row and column elec^ finger Joints and th^ center of the^^Sl ^ot sho^^^^ 

11^* P., M ^A^'^.oo ^ . 30 '^'Pf^'^lP'^^^^^^^ at aUifthe hand is not flattened 

U.5>. Pat. No. 5,463388 to Boie et al. has a capacitive o° surface. Wthout these intermediate linkages between 
sensmg system appKcable to either keyboaid or mouse fingertips and palms the overaU hand structure can only be 
mput, but does not consider the problem of integrating both S^^sed at, making hand contact identification very difficult 
typ^ of input sunultaneously. Though they mention inde- "ence the optical flow and contour tracking technioues 
pendent detecUon of arrayed unit-cell electrodes, their "^""^ ^^""^ bee° applied to free-space hand sign languaee 
capaataiice transduction circuitry appears too complex to be * recognition as in F. Quek, "Unencumbered Gestural 
economically reproduced at each electrode. Thus the long ^^^^rBctionr IEEE Multimedia, vol. 3, pp 36-47 (1996) do 
lead wires connectmg electrodes to remote signal condition- a^^^^^ss the special challenges of proximity imace track- 

mg cucuitry can pickup noise and will have significant 

capacitance compared to the finger-electrode self- ^ Synaptics Corp. has successfiUly fabricated their ele^ 
do'nS^' agam kmiting signal-to-noise mtio. Also, they ^ trode anay on flexible s^ ^^t 

t l^^ ^ T"^"^ of independent electrodes board. Tlus is suitable for conforming toihT<^^o^^f 
for muluple finger trackmg or mention how to track mul- special products, but does not provide siS^^t ^er 
up e fing.1. on an mdependent electrode array. cushioning for large surfr^ces. Even if a c^S^SpS 

ixe built an early multi-touch electrode array with 7 mm 45 the lack of stretdiability in the film leads 

by 4 mm metal electrodes arranged in 32 rows and 64 and electrodes wouU limit the compUance affoided'by the 

compresable material. Boie et al suggests that pladne 
(FMTSIDr toUl aoive area measured 12" by 16", with a fomprcssible insulators on top of the electrode array cush- 
V.U75 mm Mylar dielectric to insulate fingers from dec- ^^^^ finger impact. However, an insulator more than about 
irodes. badi electrode had one diode connected to a row 50 ™i^*^eter thick would seriously attenuate the measured 
cna^ Ime and a second diode connected to a column fingernelectrode capacitances. TTius there exists a need in the 
aist^aigmg hne. Electrode capacitance changes were mea- art fi3r a method to transfer finger capacitance influences 
sured smgly or m rectangular groups by raisii^ the voltage through an arbitrarily thick cushion 
on one or more row lines, selectively charging the electrodes 

ID those rows, and then timing the discharge of selected « SUMMARY OF THE INVENTION 

All of fh,. •. 60 It K also an object of the present invention to provide a 

,„:^r. , f «nsmg art cited above system and method for distinguishing different W^of 

utihze mterpolation between electrodes to achieve high manual input such as typing ^tiTfe de^f 

raLTcnr ""rf^ ''""^ '^'^'y- ^ --iPuli and hand,^^ on^^SuS^t via 
c^ntrl frZ n^'" *i «"nputation of a different hand confiomatio^which are easy for^ ,Sr to 

^ ^. ?Z "^^^^ '"««"g«- «S ''"^ '^'^ system to recognize 

^T£:^'tit'Z^:^'^^{'^^^^ . '"^^r^-^Je-f^presentinontoprovidean 
y arouna local maxima to inchide unproved capacitance-transducing apparatus that is die^ly 
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implemented n^ each electrode so thai two-dimensional an output node; a dielectric-covered sensing electrode con- 

seiKor arrays of arbitrary size and resohition can be built nected to the common node between the two switching 

without degradation m signal to noise. means; a power supply providing an approximately constant 

It is a further object of the present invention to provide an volUge connected to the input node of the series-connected 

electronic system which minimizes the number of sensing 5 switching means; and an integrating current-to-voltage 

electrodes necessary to obtain proximity imagps with such translation device connected to the ou^ut node of the series 

resolution that a variety of hand configurations can be <^nnected switdiing means, the current-to-voltage transla- 

distinguished. tion device producing a voltage representing the magm'tude 

Yet another objed of the present invention is to provide a ''^i!'^ self-capacitance of the sensing device. 

mulU-touch surface apparatus which is compHant and con- achieve the objects, the present invention 

toured to be comfortable and ergonomic under extended use co'^Pnses a multi-touch surface apparatus for detecting a 

Yet another object of the present invention is to provide ^^ITT^.I^^^^^^ "^"f" "^""^"^ 
tactile keyorhandpositionf^ackwithoutimpedinghand Ll^^^^^ apparatus comprising: one of a 
resting on the surface or smooth, accurate slidinj^ the ^^J', ^f^^.u ' ^/^^ of two^hmensional 
surface ^ w=^mc arrays of one of the sensmg devices (recited in the previous 
„ - ^. . . ^ . . . paragraph) arranged on the surface in groups wherein the 
Itisa furtherobjedof thepresentmvention to sensing devices within a group have their output nodes 
elertronic system wtach can provide miages of flesh prox- connected together and share the same integrating capacitor 
Z>tv nf'^ f "^T"^ "^"^^^^ resolution that a charge depletion switch, and voltage-to-voltage translation 
variety of hand configurations can be distmguished. ^ circuitry; control circuitry for enabling a single sensor 
It is another object of the present invention to provide an device fiom each two-dimenaonal array; means for select- 
improved method for invoking cursor motion continuation ing the sensor voltage daU from each two-dimensional 
only when the user wants it by not invoking it when array; voltage measurement circuitry to convert sensor volt- 
significant deceleration is detected. age data to a digital code; aixl circuitry for communicating 
Another object of the present invention is to identify 25 digital code to another electronic device. The sensor 
different hand parts as they contact the surface so that a yoltagp data selecting means comprises one of a multiplex- 
variety of hand configurations can be recognized and used to circuitry and a plurality of voltage measurement circuits, 
distinguish different kinds of input activity. To still further achieve the objects, the present invention 
Yet another object of the present invention is to reliably comprises a multi-touch surface apparatus for sensing 
extract rotation and scaling as well as translation degrees of ^ diverse configurations and activities of touch devices and 
fteedom from the motion of two or more hand conUcts to aid generating integrated manual input to one of an electronic or 
in navigation and manipulation of two-dimensional elec- electro-mechanical device, the apparatus comprising: an 
tFonic documents. array of one of the proximity sensing devices described 
It is a fiirther object of the present invention to refiablv ^^7^' ^ dielectric cover having symbols printed thereon 
extract tilt and roU degrees of freedom from hand pressure ^^"^ action-to-be-taken when engaged by the touch 
differences to aid in navigation and manipulation of thi^- scannmg me^s for forming digital proximity 
dimensional environments. images trom the array of sensmg devices; caHbrating means 

mvention wdl be reaLzed and attained by means of the motions; and communication means for "^riT^ 

^Sed^W particularly pointed out in the signals to the electronic or electro-mech^^dtw.^.^ 

T« .h. ^ .A- L *^ To even further achieve the objects, the present invention 

nfl^ 7 ^ ^.""^"^."'"^'^'P"^* rij^ ^ multi-touch surface apparatus for sensing 

of the mvenlion. as embodied and broadly described herein. diverse configurations and activities offingens and palms^f 

?htr""'^°° """P"^ * 'l*^ ^ sensitive to one or more hands near the surface and generating integrated 

n^^J" f^-^P''?^. '^gb' '>y •changes in mamial input to one of an electronic or electro-mechlnical 
projomity of a touch device to the sensii^ device, the so device, the apparatus comprising: an array of proximity 

f?"" comprising: two ekclrical switching means sensing means embedded in the surface; scanning means for 

a common node, an forming digital proximity images from theproZities mea- 

Z^J^ ' ^^J".*"! ' " dielectnc-covered sensing sured by the sensing means; image segmentation means for 

ekctrode connected to the common node between the two coUecting into groups those pr^^age pixels inten- 

switching means; a power supply providing an approxi- ss sifiedbycontactof thesamedisUnguishable partofa hand- 

mately constant voltage connected to the input node of the contact tracking means for paraSzing^^d conTacI 

senes^nnecled switchmg means; an integrating capacitor features and trajectories as the contacts move across suc- 

L'^^^fn'/n^ transferred during multiple consecu- cessive proximity images: contact identification means for 

Uve ^vtfchmg of the senes connected switchmg means; determining which hand and which part of the hand is 

another switching means oomiected in paralfel across the 60 causing each suifece contact; synchronization detection 

mtegrating capacitor to deplete its residual chaige; and a means for identifying subsets of identified contacts S 

"^'f'^h""^'' translation device connected to the output touchdown or liftoff the surfiice at approximately the same 

node of the senes^nnected switchmg means which pro- time, and for generating command dgnals in response to 

duces a volt^e represenhng the magnitude of the self- synchronous taps of multiple fingeis orihe suifacTtyping 

capacitance of the sensmg device. Alternatively, the sensing 65 recognition means for generating intended key symbok 

device composes: two electrical switching means oomiected fiom asynchronous finger Ups; motion component extiac- 

together m senes havmg a common node, an input node, and Uon means for compressing multiple degrees of freedom of 
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multq)lle fmgere into degrees of freedom common in two and imity images, the method comprising the steps of: traddnR 

three dimensional graphical manipulation; chord motion each touching hand part across succSiveprorimityimaEes- 

recogmtioo means for generatmg one of command and measuring the times when each hand part toudies down and 

cursor manipulation sign^ m refuse to motion in one or lifts ofiF the surface; detecting when band parts touch down 

more exacted degrees of freedom by a selected combina- 5 or lift oflf simultaneously; producing discrete key symbols 

uon of fingers; pen gnp detection means for recognizing when the user asynchronously taps, holds, or slides a finger 

contact amngements which resemble the configuration of on keyregions defined on the sirface;7roducing di^te 

ae band when gnppmg a pen. generating inking signals mouse button click commands, key commands, orno signals 

from motions of the mner fingpis, and generating cursor when the user synchronously Ups two or more fingers from 
manipulation signals from motions of the palms whOe the jq the same hand on the surfece; producing gesnue commands 

inner fingers are lifted; and communication means for send- or multiple ifegree-of-freedom manipulation signals when 

ing the sensed configurations and activities of finger and "ser slides two or more fingers across the sur&ce; and 

palins to one of the electronic and electro-mechanical sending the produced symbols, commands and manipulation 

device. signals as input to an electronic or an electro-mechanical 

To fiirther achieve the objects, the present invention 15 
comprises a method for tracking and identifying hand con- . ^° further achieve the objects, the present 
tacts in a sequence of proximity images in order to support indention comprises a method for choosing what kinds of 
interpretation of band configurations and activities related to signals will be generated and sent to an electronic or 
typing, multiple degree-of-freedom manipulation via ^|f?^echanical device in response to tapping or shding 
chortis. and handwriting, the method comprising the steps ^ °J Tff^ ?° * multi-tou<A surface, the method comprising 
ot segmenting each proximity image into groups of elec- « steps: identifying each contact on the surface 
trodes which indicate significant proximirj^h ™ ^^'^'t^"^' ^^^^^ P*^' measuring the times 
i^resentir^ proximity of^distin^^'h^^S^p^l^? ^T'^tlT^^^^^^t^t^^ 
oUier touch device; extracting total proximity, position, finger floating stale before any oS o "SLl^riSs baS 
shape s««. and onentation palters from each group of ^5 off the surfa^; choosing the kinds of inpS^JiS^to S 
elertrodes; backing group paths through successive prox- generated by further distinctive motion of the ^gpre from 
imity linages mchidmg <ktection of path endpoints at con- the combination of finger identities in the set; generatinng 
Uct touchdown and liflofit computii« velocity and filtered input signals of this kind when fiirther distinctive motions of 
position vectOR along each palh; assigning a hand and finger the fingers occur; forming a subset any two or more fingers 
Identity to e^h contact path by mcorporating relative path 30 which touch down synchronously after at least one fi^er 
posttions and velocUies, individual contact feahiies, and has lifted back off the surfiux; choosing a new kinds of imut 
previous estmiates of hand and finger positions; and main- signals to be generated by fiirther distinctive motion of the 
Uimng estimates of hand and finger positions from trajec- fingers from the combination of finger identities in the 
tor^ of patiis oixrentiy assi^to the fingpis. wherein the subset; generating input signals of lis new kind when 
estimates provide high level feedback to bias segmentations 35 fiirther distinctive motions of the fingers occur and con- 
and Identifications in future images. timung to form new subsets, choose and generate new kinds 
lo still turther achieve the objects, die present invention of input signals in refuse to liftoff and synchronous 
comprises a metiiod for integrally extracting multiple toudidowns until all fingers lift off the surface 
degrees of fr^m of hand motion from sUding motions of To fiirther achieve the objects, Uie present invention 
'^°''l'P°'^^S.cisoti^dictossimm-toxiAs>iTt»cc, 40 comprises a method for continuing generation of cursor 
one of the fingers preferably bemg the opposable thumb, the movement or scrolling signals from a tangential motion of a 
method compnsmg the steps of: tracking across successive touch device over a touch-sensitive input device surface 
,u P""°™"y ^ teajectories of after touch device liftoff from the surface if the touch device 
mdividual hand pats on the surface; finding an innermost operator indicates that cursor movement continuation is 
and an outermost finger contact from contacts identified as 45 desired by accelerating or failing to decelerate the tangential 
tngers on the given hand; computing a scaling velocity motion of the touch device befi>re the touch device b lifted 
component from a change m a distance between the inner- the method comprising the following steps: measuring' 
most and outennost finger contacts; computii« a rototional storing and transmitting to a computing device two or more 
velocity component from a change in a vector angle between representative tangential velocities during touch device 
the innermost and outermost finger contacts; computii« a 50 manipulation; computing and storing a liftoff velocity from 
frans ation weighting for each contacting finger, computing touch device positions immediately prior to the touch device 
franstational velocity components in two dimensfons from a liftoff comparing (be liftoff vetocity with the representative 
translation weighted average of the finger velocities tangen- tangential velocities, and entering a mode for contimiously 
tial to surface; suppresavely filteni^ components whose moving the cursor if a tai^ential liftoff direction approxi- 
spewis are conastenUy lower than the fastest components; ss mately equals the representative tangential directioiB and a 
transmitting the filtered velocity components as control tangential liftoff speed is greater than a predetermined 
signals to an electromc or electro-mechanical device. fractional multiple of representative tangential q«cds; con- 
To even fiirther achieve the objects, the present invention tinuously to^ansmitting cursor movement signals after liftoff 
comprises a manual input integration method for supporting to a computing device such that the cursor movement 
diverse hand input activities such as resting the hands, 60 velocity oorreqjonds to one of the representative tangential 
typmg, multiple degree-of-freedom manq)ulation, command velocities; and ceasing transmission of the cursor movement 
geshiring and handwritii^ on a multi-touch surface, the signals when the touch device engages the surfrice again if 
method enabling users to instantaneously switch between comparing means detects significant deceleration before 
ttie mput activities by placing their hands in different con- liftoff, or if the computing device replies that the cursor can 
figurations compnsing distinguishable combinations of rela- 65 move no farther or a window can scroll no farther 
tive hand contact timing, proximity, shape, size, poation. It is to be understood that both the foregoing general 
motion and/or identity across a succession of surface prox- description and the following detailed description ie exem- 
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FIG. 22 is a flow chart of persistent path tracking; 
BRIEF DESCRIPTION OF THE DRAWINGS RG. 23 is a flow chart of the hand part identification 

. algorithm; 

embodimentsoftheinventionandtogetherwiththedescrip- inr ^ija t * r • • 

tion serve to explain the principles of the invention In the *^*r'-^^*f*P^o'°fo"«°^tion weighting factor for right 
drawings: thumb, right mncr palm, and left outer pahn versus contact 

iTii^ 1 • ui 1 J- , . . in orientation; 

HG. lis a block diagram of the mtegrated manual input vir -y^n;.. i * f.u u • c 
apparatus; ^ *^'^-25B is a plot ofthumb size factor versus contact size; 

HG. 2 is a schematic drawing of the proximity sensor ^1^1 ^ * ^^""^ ^ ^^"^^^ ""^"^"^ 

with voltage amplifier, Proximiiy sensor contact proxmiity to contact eccentricity; 

HG. 3 is a schemalic drawing of the proximity sensor 15 ^^^^^^ ^ separation factor versus 

with integrating current amplifier P^°«">"y censor 15 distance between a contact and its nearest neighbor contact; 

HG. 4 is a schematic drawing of the proximity sensor alSh^' ' ^""^ '^^'^ ""^"^ ^""^^ "'^^'^^ 

unplemented with field effect transistors; mn • 

FIG 5i<;aQr'hem>tiV • ^' ^ a flow chart of an alternative hand part 

WO. 5 is a scheinauc drawmg of the proxunity sensor as identification algorithm; 
used to implement 2D arrays of proximity sensors; 20 r^.^ -yn - a u c ^ 

FIG. 6 is a block diagram showing a t>iical aiSitecturc Sr' ^9 ' Z . ^.^u" ^"""^ 

fora2Darrayofproxi^tysensorsther^allse^"^^ al^on^hm identification 

the same amplifier; ^ ' 

for a 2D array of proximity sensors where sensors within a pin , f^^' , ^ 

row share the same amphfier ^ "'^ handedness factor versus vertical 

HG. 9 is a schematic of a :^cuitusefiil for enabling the 30 ""^Gsl^^^ 

output gates of aU proximity sensors within a^up „.J^1* 31C is a plot of the palm cohesion factor versus 

(arranged in cohimns)- maxunum honzontal separation between palm conUcts 

TTrr- 1ft- -J • o x*,^ within a hand; 

HG. 10 is a side view of a 2D proxumty sensor array that pir 1^ o 1 * f*u - 

is senative to the pressure exerted by nonKXjnductine touch , ;fl ^ J • ^^^"^ ^^^'^ ^^'^^ ^ 

objects; ^ 35 between the mnermost and next innermost finger 

iTT/^ tt ' -J #• contacts; 

HG.U IS a side view of a 2D proximity sensor array that Rfi \^ . r,i^* «f - ♦ u ^ 

provides a compliant surface without loss of «)atial sensi- th.^ ^ the inter-hand separaUon factor versus 

tivity; estimated distance between the right thumb and left 

HG. 12 is a side view of a 2D proximity sensor array that xi a . 

IS sensitive to both the proximity of conducting touch ^ Jj^ is a flow chart of hand motion component 

objects and to the pressure exerted by non-conducting touch ^ 17,^ « - 

objects; **1G. 35 is a diagram of typical finger trajectories when 

HG. 13 isan example proximity image of a handflattened ^"^fj*^"'^!,'^^' 
onto the surface with fingers outstretched; rlO. 36 is a flow chart of radial and angular hand velocity 

HG. 14 is an example proximity image of a hand partiaUy « 
closed with fingertqjs normal to surface* MO. 37 is a flow chart showing extraction of translational 

HG. 15 is an example proximity image of a hand in the ^"^^^''^S'^ cornponents; 
pen grip configuration with thumb and index fineers ^ is a flow chart of differential hand pressure 

pinched; ^ extraction; 

HG. 16 is a data flow diagram of the hand tracking and ^ ^ ^ ^S^"^ synchronization 

contact identification system; detection loop; 

HG. 17 is a flow chart of hand position estimation; ^ ^ °^ '^P detection; 

HG. 18 is a data flow diagram of proximity i^age "WA is a flow chart of the chord motion recognition 

segmentation; 55 ^*^Pi 

HG. 19 is a diagram of the boundary search pattern • ^OB is a flow chart of cAord motion event genera- 
during construction of an electrode group; 

HG. 20A is a diagram of the segmentation strictness ^ * ^"""^ ^^^"""^ moiphing; 

regions with both hands m their neutral, default position on 42 is a flow diart of the keypress detection loop; 

surface; ^ HG. 43A is a flow chart of the keypress acceptance and 

HG. 20B is a diagram of the segmentation strictness ^^"^^-^ ^^op; ^ 
regions when the hands are in asymmetric positions on ^ * typematic emulation. 

DESCRIPTION OF THE PREFERRED 

HG. 20c IS a diagram of the segmentation strictness 65 EMBODIMENTS 

ujc icii nana is on me surtace, preferred embodiments of the invention, examples of which 
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FIG lis a system block diagram of the entire integrated '^ter. luimer aescnbed 

manual mput apparatus. Sensors embedded in the multi- 5 The host communication interface keens events from ho.h 
S'r.^Th' t"^ proximity of entire flattened hands4, "he "yping recognizer 12 and chord mS r^ J 
hngertips. thuml^ pahns. and other conductive touch » s*^' temporally ordered queue and dispatches them to 
de^^ces to the surface 2. In a preferred embodiment, the ?^'«»»«»nP"tersystem 22. The method of communication 
surface is large enough to comfortably accommodate both "^tw^n the interface 20 and host computer system 22 can 
hands 4 and IS arched to reduce forearm pronation. lO vary widely depending on the function and piocessing 

in alternative embodiments the multi-touch surface 2 may °^ ""^ computer. Id a preferred embodiment, the 

be large enough to accommodate motion of one hand, but fommumcalion would take place over computer cables via 
may be flexible so it can be fitted to an armrest or ctothing ^T?' protocols such as Apple Desktop Bus. 

Hectronic scamung hardware 6 controls and n«ris from S ft^RW ""T^ P^"'"*' ^r PCs. or Unive,^ 
each proximity sensor of a sensor array A caKbratinn " v^^^J; ^""^ra^e embodiments the software 

module* constructs a ,BW proximity imagp from ^ri^e °^'^^^^^^°'^^P^'^'"^^^^<' 
scan of the sensor array and subtracts off any b^S S^^T ^ ?! "™lti 'ouch surface apparatus would 
sensor oflseCs. The background sensor ofeets cJSX .^.TT '"""^ •'^"^ to scan the proximity sensor 
a proximity image taken when nothiiffi is louchi™ the '.^T P""°°"'y ™»S« compress and send 

surface. ^ ujucmng ine ^ them to the host computer over a wireless network. The host 

successive imagps ^ ^ 25 In a preferred embodiment the host computer system 

rec^m^r moAile 12, finger synchronization detection screen. However, in alternative embodiments the host f«m 

•n,, ^, pressure between a touch device such as a fincer 

„ JSt. ^, '^"^^ 12 responds to quidc and the surface 2, not in reference to the distance bct^n 

f ^^'^ ^ '^•'•y "^y^ '^j'^"' "Horizontal" and "verticd^S to ?^d 

hL^Z«'^.'°«^ft!'l^*y°^°'^'^*«°°«^««°''» y''^'^o"'J«es within the surface ph^^lSuTyn,^^ 
hand.ItattemptetofindtfH.keyregionnearesttothek«,tion surements are then inteipreted as preJure in a^So^ 
of each finger tap and fonvards the key symbols or com- to the surface. The dikction "Ler^ mTaL ^^Ac 
mands associated with the nearest key region to the com- thumb of a given hand, and the directfon^ute^ m«m! 
munica^n mter^ module 20. towanls the pinky finger of a given hanS Fort^^u^ 

Ite hnger syndironization detector 14 checks the fingpr «> this descr^tion. the thumb is considered a finger unless 
actmtywithmahandfiarsimultaneouspiessesorreleasesof otherwise noted, but it does not count as a fingertip "Con- 
a subset of fingprs. When such amnltaneous activity is is used as a general term for a hand part when it touches 

detected. It signals the typing recognizer to ignore or cancel appeals in the current proximity image and 

Keystrote piocessmg for fingeis contained in the synchro- ""^ 8™"P Path data structures which represent it 
nous subset. It also passes on the oombinatioD of firmer FIG. 2 is a schematic diagram of a device that outputs a 
Identities in the synchronous subset to the cfaoid motion ^o'^age « dependent on the proximity of a toudi device 38 

» conductive sense electrode 33. The proximity sensing 
nie motion component extraction module 16 computes includes two electrical switching means 30 and 31 

multiple degrees of freedom of control from individual fot^nected together in series having a common node 48 an 
finger motions during eaaly peiformable hand manmula- ^ "^^^ node 46, and an output node 45. A diin dielcrtric 
tons on the surface 2, such as hand trandations, hand """tenal 32 covers the sensing electrode 33 that is electri- 
rotatmn about the wrist, hand scaliig ^gra^wi^ <»i»nected to the common node 4«: A power supply 34 

lingers, and differential hand tflting. providing an proximately constant voltage is connected 

The chord motion recognizer piodnces chord tao or „ '*'^'"eferenoe ground and the input node 46. The two 
motion events dependent upon both the synchronized finger f."^"^^ switches 30 and 31 gate the flow of chaige from 
subset identified by the synchronization detector 14 andon . ^'^^ 34 to an integrating capacitor 37. The 

the direction and speed of motion extracted in 16 These mtegrating capacitor 37 is translated to 

events are then posted to the host cwmnunicatioD inleiface If*^^ ■ ''^ * ^igh-impedance voltage amplifier 

20. 35. The plates of the integrating capacitor 37 can be dis- 

Tbc pen grip detection module 17 checks for ««rifi. " charged by closing electrical switch 36 unta the voltage 
arrangemenfo? identified iZSiiLtS^ :S^S«»3rr ''/Z'^.'^Z- -^^ 

hand IS configured as if gripping a pen If such M^TnoT k . 30 and 31 are opened and dosed in sequence 

ment is detected, it forwKlf mJSTe J^^t^ ^ „^ "^"^ ^"gh they may be 

fingensasinkingeventsto the host communicati^S^ 6S % a .^^ ^ ^ 2- ^'^'^ 

20. These inking events can either lay di^^^^^ " f referred to as the mput switch; electrical switch 

computerdisplay fordrawingorsigJlXte™^ 3^^ ^ '^^HZ'^Z^U^ ^'^'^ 
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1116 proxumiy sensing device shown in HG. 2 is operated when its gate voltage is logic 0. A controUer 42 is used to 
by closing andopemngtheelectncalswitches30.31.and3« apply gate volUges as a function of time as shown in FIG 
in a paitiailar «quence after which the voltage output fiom 4B. In this example, a sequence of three paiis of pulses (43 
SLr£ « M °" ^""^^ °^ * and 44) axe appUed to the input and outpiit txansikor gates, 

touch device 38^ recorded. Sensor operation begins with S Each pair of pulses 43 and 44 product a voltage cLnge 
S,Z^^..h f ^, T ^ ^ intoff^ting capacitor 37 as shown in^tion 2. 

shorhng swit^36 is then closed for at sufficienUy long time The number of puke paiis appUed to input 43 and output 44 
^ I^w .^f?{PT!?"^ °° mtegrating capacitor 37 gates depends on the desired voltage across inte^ating 
to a low eveL The shorting switch 37 is then opened. The capacitor 37. In typical appUcations tte number is b^e^ 
input switch 30 is then closed thus aUowing charge to flow lo one and several hundred pulse-pairs 
betw^n the power supply and the common node 48 mitil the FIG. 5 shows the proximity sensor dicuitry appropriate 

^Sl^^i T ^"^^ ^ ^"^^ ^ ^ » comprising'an array of proriiS seLor^ 

will accumulate on the sensmg electrode 33 according to 47 as in a multi-touch surface system. TTie proximity sensor 

Q-v(i!'A)/D '•7oonsistsoftheinputtransislor30,theoutputtransistor31, 

I J 15 the sensing electrode 33, the dielectric cover 32 for the 
where V is the voltage of the power supply 34, e is the sensing electrode 33, and conductive traces 43, 44, 45, and 
permittivity of the dielectric sensing electrode cowr K and 1*^ conductive traces are arranged so as to allow the 

the air gap between the cover and the touch device 38, D is P">ximity sensors 47 comprising a 2D array to be closely 
the thickness of this dielectric region, and A is the overlap Packed and to share the same conductive traces, thus reduc- 
area of the touch device 38 and the sensing electrode 33. ^ •be number of wires needed in a system. FIG. 6 shows 
Therefore, the amount of charge accumulating on the sens^ *° example of such a system where the input nodes 46 of all 
ing electrode 33 will depend, among other things, on the area proximity sensors are connected together and connected to 
of overlap of the touch device 38 and the senang electrode * power supply 34. The output nodes 45 of all proximity 
33 and the distance between the touch device 38 and the sensors are connected together and connected to a angle 
sensing electrode 33. The input switch 30 is opened after the " >°'*^graling capacitor 37, a single shorting transistor 36, and 
voltage across it has become zero, or nearly so. Soon after ? single voltage-to-voltage amplifier 35. In this 
input switch 30 is opened the output swit(^ 31 is closed until "uplemenUtion, a sii^e proximity sensor 47 is enabled at 
the voltage across it is nearly zero. Qoang the output switch * '''' W'ying a logic 1 signal first to its input gate 43 
31 allows charge to flow between the sensii^ electrode 33 ^^"^ '° ou^ut gate 44. This gating of a single 

and the integrating capacitor 37 resulting in a voltage change ^ proximity sensor 47 one at a time is done by input gate 
across the integrating capacitor 37 according to: controUer 50 and ou^ut gate controller 51. For example, to 

enable the proximity sensor 47 in the lower right coner the 
delta v=<v-Vc)/(i4C«D/e'A) (2) input gate controUer 50 would output a logic one pulse on 

. u • .K. 1. .. • conductive trace 43a. This is followed by a logic one pulse 

where Vc is the voltag^ across the mtegraUng capacitor 37 35 on conductive trace 44* produced by output Ste controller 
before the ou^t switch 31 was clewed, C is the capacitance 51. Repetition of this pulse as sho^^ in HG. 4B wouldl 

vL^rw^r?™7™t^ r "P ^'^^'^^ capacitor 37 and a 

F^r^„r^^ mT ^.f '^'-'^ "^"^ '° corresponding voltage to appear at the output of the ampU- 

hi- 7 * ^^^^^ °^ ^ """P"' ^- '^'^ of proximity seLors 47 is thus 

31 switches as described above produce a voltage on the « scamied by enabling a single sensor at a time and recording 

mtegraUngcapacitor37thatreflectstheproximityofatouch its output aauieconung 

dev.ce 38 to thesensing electrode 33 eC. 7A is a schematic of typical circuitry useful for 

»h K ".K Pronmity sensor in converting the proximity sensor output 58 to a digital code 

^ "^V^^ voltage-to-voltage appropriate fi,r pnKxssing by computer. THeToximity 

fra^kuon device 35 are replaced by a resistor 40 and a 45 sensor output 58 is typicaUy non-zero even when ttere is ni 

cunent-to-voltage Uanslation device 41, respectively. The touch device (e.g., ref. no. 38 in FIG. 2) nearby. TTiis 

mtegrating func^on of capaator 37 shown in RG. 2 is. in non-zero signal is due to parasitic or striy capi^it^« 

this vanauon of the proximity sensor, carried out by the present at the common node 48 of the proximity sensor and 

capaator 39 shown in nc. 3A. THose skiUed in the art wiU of relatively constant value. ltlSS;e3e ^ 

^„ 1 P^f-^y produces a 5o non-zero background signal before converting the sensor 

^ i°'^'^'\^?^^°^*^^P"' """P^' 58 to a digital code. This is done by using a 

^r,*^ T!! dependmg on therelative value of the differential amplifier 64 to subtract a stored re«ird 6f the 

r^^ T' t^To /^^'^'^l ^"^^ "^^"^ background signal 68 fiom the sensor output 58. llie result- 

A^TIMT^ f ^^J?\ "°P"^" "^""^'y difference signal 65 is then converted to a digital code by 

Al^ough, the circuits shown m FIG. 3 provide a more linear 55 an ADC (analog to digital converter) 60 piodudne a K-bit 

output than the circmt shown in FIG. 2 the circuits of FIG. code 66. Tlie s^red bLground sgi^S^^^ by 

3 generally require dual power supplies whfle the circmt of sampUng the array of proximity sensors 47 (FIG. 6) with no 

na 2 requires only one. touch devices neaiby and storing a digital code specific for 

■ae electrical switches shown m FIG. 2 can be imple- each proximity se4r 47 in a meifory devi Tte 

.TJ^«r'.h- T°"fh-!JfT'°' « particular code corresponding to the background signal of 

mtegrated, thm film, thidc film, polymer, optical, etc. One each proximity sensor is sefccted by an IVllit addreTinput 

^r^iZ!' IS shown m FIG. 4Awhere fieU eflfect 70 to the memory device 63 and appUed 69 to a DAC (dSul 

transistors (FETs) are used as the input 30, output 31, and to analog converter) 61 

shorlmg 36 switch«^ Tlie FETs are switched on and off by TTie 2D array ofproximity sensors 47 shown in FIG. 6can 

voltages applied to their gate terminals (43, 44, and 55). For « be comiected in groups so as to improve the rate at which the 

the purpose of this description we will assume the FET is entire array is scanned. This is ilhislrated in FIG. 8 where the 

switched on when its gate voluge is logic 1 and switched oflF groups are arranged as columns of proximity sensors. In this 
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m Pir #: Tkl ^ a Power supply 34, as sensing is relaUve to this surface. The electrodes W anH 

prn i:> cwT f''°P"^^ «irface milhmeters of the surface show up at all. 
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Z^hfL'^^T'""^ ^ ^'"^ ^'"^^y ootationaJ puiposes identified paths 

top of the fingers actuaUy LcHTs^^ Sldo7 S e s ^^'I^^' ^ ^S*^^ 204, F5 for the 

fingertips.T^curlingJderofU^^^ ' PHiky finger 2(^F6 for the outer palm heel 206. F7 for tte 

them behind the pinXl thumb 201^SreSS^ TL^ ""^'^^ Z ^ '^"^^ ^^"^ 

very close to the palm heels 206, 207. TTie knuckles also « ^ i^?^"^ ^ "^'^^^^ ^ ^ 

appear larger than\he curled fingirtips of U tot^ T T^^ifS.^^^' ^ 

same size as the flattened fingertiiTin FIG 13 Sie^ lo ^^^.^P**^' ^.^^^"^^ right index finger path. W^^^ 

differences in size and arrangement^ be meas^bT^^ ^3 ^HS'^r'^K.''^^^^^ 

pen grip detector 17 to d^guish this peo grip conC- t ft h H ^ r*"""?^ ^ ^"""^ ^^^"^^ 

tion from the closed and flattened hand^nW^^ if hand rdenUty is represent*^ by a -1 and right hand by 

FIG. 16 represents the data flow within the contact ,1' ^Tk^ ""^T "^^^ ^'^^^ measurements 

tracking and identification module 10. The image segmen- is ^ "T" °^ symmetry. 

Ution process 241 takes the most recently scanned proximity " is also convenient to mamuin for eadi hand a set of 

image data 240 and segments it into groi^ of electrodes 242 ^^^^^^ data registers for which each bit represents 

corresponding to the distingmshabie hand parts of FIG. 13 touchdown, continued contact, or liftoff of a particular 

The filtering and segmentation rules qjpUed in particular Bit positions within each bit field correspond to the 

regions of the image are partially determined by feedback of 20 indices above. Such registers can quickly be tested 

the estimated hand ofifeet data 252. The image segmentation ^ ™^ ^ determine whether a particular subset of 

process 241 ou^uts a set of electrode group data structures has touched down. Alternatively, they can be fed into 

242 which are parameterized by fitting an ellipse to the ^ lookup table to find the input events associated with a 

portions and proximity measurements of the electrodes P^'^'^ fi^gcr chord (combination of fingers). Such finger 

within each group. 25 ^^^°^^^y ^'itfields are needed primarily by the synchroniza- 

The path traddng process 245 matches up the parameter- ^tector 14 and chord motion recognizer 18. 

ized electrode groups 242 with the predicted continuations process within the tracking and identification 

of conUct path daU structures 243 extracted from previous subsystem is the hand position estimator 251, whidi as 

images. Sudi path tracking ensures continuity of contact described above provides biasing feedback to the identifi- 

representation across proximity images. This makes it pos- 30 ^d segmentation processes. The hand position esti- 

sible to measure the velocity of individual hand contacts and ^ intended to provide a conservative guess 252 of 

determine when a hand part lifts off the surface, disappear- lateral hand position under all conditions including when the 

ing from fiiture images. The path tracking process 245 ^^d is floating above the surface without touching. In this 

updates the path positions, velocities, and contact geometry estimate represents a best guess of where the hand 
features from the parameters of the current groups 242 and 35 ^ ^"^^ down again. When parts of a hand are touching 

passes them on to the contact identification processes 247 surface, the estimate combines the current position 

and 248. For notational purposes^ groups and unidentified measurements of currently identified hand parts with past 

paths wiU be referred to by daU structure names of the form estimates which may have been made from more or less 

Gi and Pi respectively, where the indices i are arbitrary reliable identifications. 

except for the nuU groiq) GO and miU path PO. Particular 40 The simplest but inferior method of obtaining a hand 

group and path parameters will be denoted by subscripts to position measurement would be to average the positions of 

these structure names and image scan cycles wiU be denoted aU the hand's contacts regardless of identity If hand parts 

by l^adteted mdices, so that, for example, PZJn] represents 201-207 were aU touching the surface as in HG 13 the 

the horizontal position of path P2 in the current proximity resulting centroid would be a decent estimate, lying some- 
image, and PZJn-l] represents the position in the previous 45 where under the center of the palm since the fingers and 

proxumty miage. The conUct identification system is hier- palm heels typically form a ring around the center of the 

archically ^lit mto a hand identification process 247 and palm. However, consider when only one hand contact is 

withm-hand fi^ger and palm identification process 248. available for the average. The estimate would assume the 

Ciiven a hand identification for each contact, the finger and hand center is at the posiUon of this lone contact, but if the 
palm identification process 248 utilizes combinatorial opti- 50 contact is from the right thumb the hand center would 

nmauon and fiizzy pattern recognition tediniques to iden- actually be 4-^ cm to the right, or if the contact is from a 

Uty the part of the hand causing each surface contact. palm heel the hand center is actually 4-6 cm higher or if the 

Feedback of the estimated hand oflfeet helps identify hand lone contact is from the middle finger the hand center 

contacts when so few contacts appear in the image that the unsound actually be actually 4-^ cm lower 

itie hand identificaUon proo^ 247 ulilms a separate estimator 251. The steps must be repeated for eachhand 

comb,nalona^optim|zal.oo algorithm to find the assignment separately. In a preterr^ embodimerr^p^L^utS 

r„ thl°„;^H 1^'°"^"^''*™°'^'^ within-hand »ntact identificatio^S^cotS 

m the mo^ biomecham«Uy consistent within-hand idenU- (step 254) for each contact an offset betw^ the^^S 
fications It al» receives feedback of the csto^ «, «nUct p«ition (FUnl^iTn]) ^ tteSTp^^^ 

finger ofeets 252. pnmanly for the purpose of temporarily the parti^ finger or p^tJS^fSli? )^^^^ 

Sft off^ mwsunxl hand position after fingers in a part Lntityi. Tie defaul^^Siti^^&l^r^S^^^^ 

haadMtoffthesurfa«.Thenrfthefinge,ssoontouchback finger and palm positions when the hand UTS 

S^J^LXdt''^"""'*''^'^"*'*^^ P^*^ '^'^ fing^parti^y closed, as when r^g^ 

previous baiKl ^entificaUwis. « home row of a keyboard. Step 255 averaees the individual 

■Ileou^,utof the Kl»Ufi<^uon processes 247 and 248 is conUct offi^ts to obtain a r^Lured h^ ofeet mTn] 

the set of contact paths with non-zrao ham! and finger H^n]): ^"»«J.°J. 
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^ touching the surface, H^n] is made small enough 
y Fi^n]in,[n] - ru^) estimated oflfeets gradually decay to zero at about the 

^ ^ ^ rale as a hand lazily returns to default position. 

w When H^n] is made small due to low identification 

h *^^""^ 5 confidence, the filter tracking delay becomes laige enough to 

lag behind a pair of quickly moving fingers by several 
YFL^nMR fni-Fz^^ Centimeters. The purpose of the filler is to react slowly to 

' ^ questionable changes in contact identity, not to smooth 

"^^"^ 1^ contact motion. This motion tracking delay can be safely 

E'Wh] 10 eliminated by adding the conuct motion measured between 

images to the old ofl&el estimate. Step 257 obtains motion 

Prpfpr^hlv ^ • i:- r i r . . ^™ ^^^t^, (H^nlH^nJ), of the current contact 

Preferably the weighting Fi^n] of each finger and palm velocities: 

heel is approximately its measured total proximity, i.e. 

^KuJi^h^^L^]' This ensures that lifted fingers, whose 15 hn 

proximity is zero, have no influence on the average, and that ^ ''Wnl/^^M 

contacts with lower than normal proximity, whose measured /<^[„] = tl 

positions and identities are less accurate, have low influence. j^' Fw^W 

Furthermore, if palm heels are touching, their large toUl 

proximities will donunate the average. This is beneficial 20 i=7 

because the palm heels, being immobile relative to the hand Y FLoJin\Fi^{n] 

center compared to the highly flexible fingers, supply a more « r„, ^ ±L 

reliable indication of overall hand position. When a hand is 

not touching the surface, i.e. when aU proximities are zero, ^» 
the measured offisets are set to zero. This will cause the 25 

filtered hand position estimate below to decav toward the * . t • ■ ^ , 

default hand position. ^ J^*= "'"^^ velocities, (Fi^nlFi^nJ), are retrieved 

As long as the contact identifications are correct this hand trackmg process 245, which measures them 

position measurement method eliminates i^^^^^ I^^^h T ^ ^f^' fT^' fiF 
caused by assuming lone contacts originate ftomAe center 30 

of the hand. Flexing of fingers from their default poations * 

will not perturb the measured centroid more than a couple ^nhHTn^ «JnWi-HJnDfH fn-iuw jnUt^ m 

centimeters. However, this scheme is susceptible to contact i»-«l"Ki H»JnD(H^n-iKH_[o]At) (7) 

nusidentification, whidi can cause centroid measurement "«J°>"cJn]H„4nWi-H«[nl)(H^n-i>H^o]At) (8) 

errors of up to 8 cm if only one hand part is toudiing 35 n 

Therefore, the current measured oflkets are not used dir^tly **inally, to provide a sunilarly conservative estimate of the 

but are averaged with previous ofl&et estimates (H^n-ll' Positions of particular fingers, step 259 computes individual 

H«j:n-1]) using a simple fiist^rder autoregressive filter o^^ts (Fi^nlFi^nJ) from the distance between 

forming current offSset estimates (H^n],H^n]) identified contacts and their corresponding default finger 

Step 256 adjusts the filter pole H ^nfaccording to 40 P^^^^ ^« estimated hand offeets. For each identifi- 

confidence in the current contact identifications. Since finger contra 1, the oflEsets are computed as: 

identifications accumulate reliability as more narts of the ir r i_u r t... r ^ 

hand contact the surface, one sin.ple measure oS,tffiS! '^tjIkj-V'^^"^ ■^"^''^'■•^"'^'l. 
tioo confidence is the number of fingers wfaich have touciied 

down firom the hand since the hand last left the surface 45 ^'yl°y^'M^<'hB^a}*mjinyFi^^ 

Contacts with large total proximities also improve identifi- [»-»>RJi>)At) (lo) 

f^tL""!^'^ •'^T ^""^ disambiguating These finger offsets reflect deviations of finger flexion and 

features such as size and onenUtion. TTwrefore H^n] is set extension &om the neutral posture. If the^r the 
rougMprof«monal to the maximmn finger count ph« the fingers in an ext«me conC^n such ^the fl^neS 

Z^L rr".!l*t ^ "^^"^ """^ hand configuration, the collSmagnSS^oftSel fiirr 

course be nomjahzed to be between zero and one or the filter oflfeets can be used as an indication^f user hand^?^ 

wiirbe mB^Ie.m« when confiden»^^ ^ ^ hand sm. and 

calioi^is high, Le. when many parts of the hand firmly touch The parameter (H^rnm^TnL and rFi_JnlFi fnh 

the surface, Ihe autoregressive filterfavon. the current offset fi,r each hand and L^ciSe thTlti^&lS 

^vT^^a"^ r^;; ^r^^'t' ""^""^ fi-gerofeetdaU252,ihichisfedba^to2lSme"L^ 

have reap^aied since hand LftoEt the filter emphasizes and identification processes during analysis of the nert 

^e^r^^^ffJ^ ?^ • • absolute coordinates, they can simply add (step 260) the 

esttoate S^„^^.°"tLf°.K'"rir % ^""l o&ets to the defUt finger'^iiUonL. bit in many 

STrf^^ l 'r'T'"*^"^''"^'*^'"*"^ above 60 cases the reUtive offeet repres^nUd^ is actually mo« 
the surface fiar optunal scgmentaUon and identification as convenieat a«"aiiy more 

^.SHHle'^r f 'f » hand lifts off the surfece It should be clear to those skilled in the art that many 

mthemddleofaoompexsequenceofoperationsandmust improvements can be made to the above hand p«W^n 

r^re1tMfZ'^'^'''^ff^''"''°'"''^^'^ estimationproceduiewhichremainweS^to^^Tf 

ended, the hand js likely to eventuaUy return to the neutral position of liftedhands. One improvement is^2c the 

posture, ordefault posiUon, to rest Tlierefon^, while a hand estimated hand offi^ts decay towLTro" a ^St^f^iS 
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deat as described so far. I. is actoaUy^X^- to" ^t^'^Z^^^'^r ."^ "^ ^f^"- 
impose a minimum horizontal separation between Zlui . i 1 J"^"' fingertips aren't joined into a 

o^t^'s^^f !bJ^3^t^et Sr^df^e? 2: ji^^"^™" ^» ^^^'^ of these seg- 

«timatedpositionoftheo.herhandisd5.Sn^;l de^^T,"^T '^^ ^^'^^ 

^eestoatedpositionof theliftedleftZl^ulibSo^ iTe2emof'S'^^''r'?"" ^^^"'^ 

&om default to the far left of the surface, possibly off^ ,n whii! "'"PP^ segmenUUon regions 

surface compktely. If the right hand is lif^dXftk ^ ^ " '^^'^^ 

nou an equation like the fXwinTc^rfXt W S^i^ .^l^ ""^ ^ 

the estimated right hand position out of the way ^ ? P"^'^™ °f '^^h A-g" 

f"" P^™ beel u, eadi hand. Rectan^ar outlines in the 
'*^f*-^("'<J»MLn^-Rn^u^,,^i,,_^^_ lower comers represent the left 284 and right 286 sloppy 

(11) " fegmenlation regions, where partial minima are lai^ly 

where (LFl^^-RFl^ is the default separation between Znl!L '^'^^ 1^°° "="^™°g the strict seg- 

left and right thumbsTirihe minimum hoX^ LSfon P"""""'^ P°'°«=' 

to be imposed, and LH^n] is the cJrem^taa^oS Z^v boundaries. As a preferred embodiment the 

of the left hand. esiimaied oUset sloppy regions are rectangular, their inner boundaries 285 

FIG. 18 represents the daU flow within the proximity ^^^^^ff'^^T^r'^^^'''^'^^'^'''^S^'^ 

«nagesegmentationpiocess241.Step262makesa»S Tj^ ^^ed to he and the upper boundaries 287 are 

smoothed copy 2.3 of the aurent p^xiS^^^e^^K S^te 2of T^Tf ^^^^^^ 'Z'"" °^ "^'i- 

passmg a two^nsional diffusion operator m G«^ia^ ZZ„ """i ^ boundaries of the sloppy 
kemeloveriLStep264searchesthesmSi.^ge^rr 25 ^°^,tr't'*^'^'^.^'^°^^^^^&^oia^s^Z 

localmaximumpixels2«5whosefilteredp.Sge^^' S s^^^^e^^ "^"^ "^-^^ ''"'^ 

a sigmficance threshold and exceeds the mtered pro:^« ^ f k ^"^^ segmenUhon regions return to the 

of nearest neighbor pixels. The sm^^^^T "^L^ ^ '""^ =*'y«» o^ ^^e surface 

chanoethatanisolatednoisespikeonasiSel^^ 20^s^°^ ""^^ P°^'^°" "^ff- F'G- 

result m a local maximum which exceedfthe Sca^ 30 2^^V u^^^ "^""^ '^'^ 

threshold, and consolidates local maxima to^KeTr J^^'^P^r'ff^'.^lb-ngl't hand moves toward the upper 

distinguishable fleshy contact P^ "o^^ towaid the lower left. Itis 

Process 268 then constructs a group of electrodes or t'^T* . ^ P*^ ^ ""^^ ""^ P^ ^ 'be sloppy 

pixelswhichregistersigmficantpro^i,J ^"^^"^ ^T'.T^'' ^ 

maximum pixel by searching outward ftom eaTtoS 35 left^S'.^.rf^?' f"" 'f** l^PP^ "'Sion 284 is moved 

maximum for contact edges. Each electrode eno^unte^ '''"off 'he surface entirely when the left hand is lifted off the 

betore reaching a comJ^^^T^ toTS S« .^^^^ -""^ 'o the left side of the 

maximum's gnwp. HG. 19 shows ttie basic boL^ elT '^^ ^^'^ °^ ""^ 'Altering 

trodesearchpatternforanexampleconuctbomir^4 In Se.^°^^ segmentahon region of the opposite hand. 11,1 

this diagram, an electrode or image pixel lies at the tin of ^ !l mrlemented by imposing a minimum horizontal 

each arrow. TT,eseareh starts at LE n^^^b^f ^'^P^ ""^""^ ^ 

276. proceeds to the left pixels 277 unffl J^Zf "^"^ *°°*"' ^"^^ hand with the 

isdetected.Tlielastpixelbeforethebound^2rfemS h^^T "^e estimated position of the 

as an edge pixel, and the search resumes tottifS^S ^^tZ , T"^!"^' ^ '^bart of 

the local maximum pixel 276. Once the left ^dS^L 4, ^ are applied at each searched electrode 

of the local maximum's row have been foiS^ 1^^? °" ^"."^'i!^ a strict or sloppy 

recurses to the rows above and below alwaSr^ mY^„ ^SmentaUon region. Decision diamond 290 checks wheSer 
the column of the pixel in the preWoiS row ShS th^ J«"^<«tbed proximity of the electrode is greater than the 
greatest proximity. As the exa^Ie iU^^fr^tSo L^Sm iT^*^ '^^ «l«=tnxie is labeled 

set of pixels or electrodes is a^ected in ih^maSSd 50 ^J^f^^''^^^^"- step 304 regardless of the segmenUtion 
sense but need not be rectangular. TOs aWZ^tJ ^ ,'!^r '*'^°"''^'*^'''*=P^''^=*arch returns 
ctosely fit the typical oval^ha^ of t^^ZT^"^ ^ '^'^ direction. If the 

leaving electrodes out or ind\ulingX« fo^aSSn T^^" P^T"^ '^'^^ ''^^ ^ applied 
contacts. ^ °* '^J*""' the smoothed proximity of neighboring electrodes 

If contacts were small and ahvays well senaraterf « on whether decision diamond 292 decides the 

could simply be established wher^erlXJreaSl ""if "^T^- ^ ' "^"PP^ «gion. 
fell to the background level. But somettZ IgeSaS a rlL H • """^^ ="^""=^"2 horizontally within 

onlyseparaUxibyashghtvalleyorshaUow^eSS ^P'^'°'^^'*^°°'>*'^°°''308 
To segment adjacent fingertip^ the partial ^a^TtS^ ^on^.f^.r^"""- ^ ^ " ""ri^ontal or 

valleys must be detected and used as giZ^u^di^ «, t^ ft^'^T^'^ ^'^ "^^^ ^ "^^^ «=i8h- 
Urge palm heel contacts, on die otlTr Cm^reSi S^eS^n'^ JKr'.! "T^"' "^^^^ '''i"^' 
partial minima due to minor nommiformitSn fl^ Z^- ^^^^3^^^ ^'^'^'^ «J««''«Ie is labeled 
miity across the contact If all ekctrodes under the con?^ K no. K k ^ '*^'=''0'« 305. 

are to be collected in a single group, such partd Ift .^r" """^"^ ^ "'^'^ *e row 
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ne^ ne^bor electrodes. If so, a proximity vaUey step 272 of the segmenlation process is to extract shape, size 

^Lri the thumb has probably been detected. and position parameters from each electrode group 6n>up 

^Jl^^J^^ as an e^e 312 and seaich resumes position reflects hand contact position and is^^cLanr Z 

'°°'^'^f detennine finger velocity. iW total group ^S^tv 
next row 301. If decision diamond 294 determines that as . • j - - gr""? proxumiy. 

Sloppy cegioosearchisac^an^S,^^^^ ' d'S" tr ^ 

stringent horizontal minimum^ arc performed to checl^ "^"^ Astmgmsh finger, palm, and thumb contacts, 

for the crease or proximity vaUey between the inner and Provided is the set of electrodes in group G, e^ is the 

outer pahn heels. To qualify, the electrode must be more than unsmoolhed proximity of an electrode or pixel e. and e^ and 
about 2 cm horizontal distance from the originating local lo ^ ^ coordinates on the surface of the electrode center 

maximum, as checked by decision diamond 296. Also the ^ centimeters, to give a basic indicator of group position, 
electrode must be part of a tall valley or partial horizontal proximity-weighted center, or centroid, is computed 

minim u m which extends to the rows above and below and portions and proximities of the group's electrodes: 

the next-nearest neighbors within the row, as checked by 

decision diamond 298. If so, the electrode is labeled as an 15 = V (12) 

edge 300 and search recurses in other directions 305. All '^^e 



other partial minima within the sloppy regions are ignored, 
so the search continues 302 until a background level edgp is ^' = Z ^ ^^^^ 

reached on an upcomii^ electrode. * 



In sloppy segmentation regions it is possible for groups to 20 „ 
overlap significantly because partial minima between local ^' = Z "c^ ^^^^ 

maxima don't act as boimdaries. Typically when this hap- ' 
pens the overlapping groups arc part of a largp fleshy contact 
such as a palm which, even after smoothing, has multiple 

local maxima. Two groups are defined to be overiapping if 25 ^ proximity G, integrates 

the search originating local maximum electrode of one P^ximity over each pixel in the group, it depends upon both 
group is also an clement of the other group. In the interest ^ °f * since large hand parts tend to cause 

of presenting only one group per distinguishable fleshy groups with mwe pixels, and of the proximity to or pressure 
contact to the rest of the system, step 270 of FIG. 18 surface of a hand part. 

combines overiapping groups into single supergroups before 30 Since most groi^ are convex, their shape is weU approxi- 
^f'^^T^TZ''; in the art wiU realize that mated by ellipse parameters. The ellipse fitting procedure 

requires a unitary transformation of the group covariance 
apphed m vanoi^altern^e ways to miprove the segmen- niatrix G_ of second moments G_G ,G • 
taUon process and still he well within the scc^ of this xi^"^^"yy- 
invention. For example, additional image smoothing in 35 [G^ G } 

sloppy segmentation regions could consolidate each pahn = " " \ 

heel contact into a single local maximum which would pass ^"^^ 
strict segmentation region boundary tests. Care must be r - V rr- _ 

taken with this approach however, because too much 
smoothing can cause finger pairs which une3q>ectedly enter 40 

sloppy pahn regions to be joined into one groiq). Once a G^=G^^ V e,iG^-e,y(G,-e,) (17) 

finger pair is joined, the finger identification process 248 has 
no way to tell that the fingcrt^ are actually not a angle 

palm heel, so the finger identification process will be unable = X '^f^^ 

to correct the hand position estimate or adjust the sloppy 45 '^^^ 
regions for proper segmentation of future images. 

More detailed forms of feedback than the hand position . 
estimate can be utilized as wcU. For example, the proximal eigenvalues and of the covariance matrix G^ 

phalanges (209 in FIG. O) are actually part of the finger but oetenmne the cUipse axis lengths and orientation GeC 
tend to be segmented into separate groi^ than the fii^ertips 50 

by the vertical minimum test 310. The vertical minimum test /«r^v^ (19) 

is necessary to separate the thumb group fit)m index finger- ^ . 
tip group in the partially ctosed FIG. 14 and pen grip FIG. G«*.«^>^ (20) 

15 hand configurations. However, the proximal phalanges of 

flattened fingers can be distinguished from a thumb behind 55 C« = nrutm/ ~ ^" ] (21) 

a curled fingertip by the fact that it is very difficult to flatten ^ ^ 

one long finger without flattening the other long fingprs. To 

take advanUge of this constraint, a flattened fingjer flac 267 „a ■ • , ... 

is set whenever two or more of the contacts ^nti^ as ^^"^ ^'^^'^^ "^^^ ^ ^^^^ 

index through pinky in previous images are larger than 60 convenience while distinguishing fingprt^js from 

normal, reliably indicati^ that fingertips are flattening. at higher system levels, the major and minor axis 

Then decision diamond 310 is modified during processing of '^°g^ are converted via their ratio into an eccentricity G^: 

the current image to ignore the first vertical maximum 

encountered during search of rows betow the originating c = (22) 

local minimum 276. This aUows the proximal phalanges to 65 * 
be included in the fingertip groi^> but prevents fingertip 
groups from merging with thumbs or forepalms. The last 
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Note Uiat ance the major axis lenglh is always greater than where the squared Euclidean disJance is an easily commuted 

or equal to the aunor axis length, the eccentricity wiU always disUnce metric- oJ^ance is an easUy computed 
be greater than or equal to one. Finally, the total group 

proxinuty is empiricaUy renormalized so that the typical <'^(GW1H0^-pi^'+{GVP^' (2S) 

curled fingertip will have a total proximity around one: s c.,„'i^<i,t,.„B_j r u • 

e J .vuuuuc. 3 step 324 then finds for each active path PI, the closest active 



C, := ^' (25) 

^o^ftKgtFingtt tip 



group and records the distance to it: 

PItkMsC = arg min iP(Gk, PI) Pie PA (29) 



dec 



On low resolution electrode arrays, the total g^^ Pt,,^ = ann Ack. Pi) w Pie pa (30) 
unity Cj^ IS a more reliable mdicator of contact size as well 
as finger pressure than the fitted ellipse parameters. 

SIrin; Ir'^''/- ^''''n "^^^on, the Instep326,anactivegroupGkandpathPIaiconlypaircd 

onentation and ecoentnaty of small conUcts arc set to with one another if they are closest to one another i7 

Gk.,_^andPI,_^iertooneanother^^dr^^ 

group proximity G is used as the pnmary measure of between them is less than the tracking radius AU of the 

contact aze instead of major and minor axis lengths. following conditions must hold: 

FIG. 22 shows the steps of the path tracking process, 

which chains together those groups from successive prox- ^Ktc^c^^ (3i) 

imity images which correspond to the same phyacal hand ^ PU.«c=Ok 
coritact. To determine where each hand part has moved since 

the last proximity image, the tracking process must decide "c£«-=«c«.rt<Pl«,^ (33) 

^^^u^.Tu^'.^Tgte'S ^:ie,?'gS„Ti.rpats^f - '^''^'^ " 

from the same contact will be ci;ser^ o'ne ano^r Z^o " tet^„ ^^ver IT""' fT'^'^^^ 

other groups and paths. Also, biomechanical constraS'on ^^X ^t^l^l^^.J^ " accomplished 

lateral finger velocity and acceleration limit bow far a linger S ^ lift ^ I only groups 

can travel between images. Hierefore a eroun and o^h - u P*** "^"^ 

should not be matched Less they ^e wS^a^^ deactrvaU^v«thm the last second or so d^ 

known as the tracking radius of one sS.^ ,„ ' ""^ f T has stdl not be paired with 

typicallateralseparationbetweenfin^LTSTater^Se ^iT " °LT"? '^^^^'^V'^* - '^'-ated a new 

tracking radius for reasonable imagelcan r^ toudlw^ Sce^lH^ V "^"^ \'^.- """^ 

and liftoff are easUy detected by the fact tl^ tou^Z • \ ^ '"^'^ P^*^ ^ » 

u^ally causes a ne^ group to'^^^^ olS't^tS^ 35 C^lhe^a« '"'^^^'^ "^'^""'^^ '''' 

radu of existing paths, and liftoff will leave an active path SiJIm i^^^i^. .h, ^ . ^ c . 

The fiiststep 320 predicts the current locations of surface 1^ shape (PJn], PJn]) parameters from 

contacts along existing trajectories usLS^ correspondmg group parameters, but Kahnan or finite 

velocities meLred frL ^revioS y^J^ P^^^ "^"^0^? ^ '^^^^r^* 

ous velocity to the location prediction i^ro^ ^p^^. „ fe '"^^ ^ ^ 

tionexcept whenafingersuddenlystar^^^ ^ ^t:-^"^ ^^^^ 

direction. Smce such high acceleration events occur less louows. 

often than zero acceleration events, the benefits of velocity- Pin)-G, (34) 

based prediction outweigh the potentiaUy bad predictions pr ^ 

during finger acceleration. Letting PJn-l], Pj;n-1] be the " 

position ofpathPfi^om time step n-1 and PJn-llPFn-l] ^ PJn>-G, pg. 

the last known velocity, the velocity-predicted path continu- „ r 

ation is then: i^etnj-Gg 

P^oKJn-ljMtPJn-l] (24) ^-^"^^ (3^0 

P^n}-P>-1>AtP4n-l] (25) ^-^"^ (39) 

Letting the set of paths active in the previous image be PA, Pry^nH> (40) 

and let the set electrode groups constructed in the current » r lt /a» 

image be G, step 322 finds for eacA group Gk the closest "i^J^-'^ (41) 

active path and records the distance to it else if group G is a continuation of active path P[n-1] to time 

60 °- 

CiU«P = arBminrf^(C*.PO VCteC (26) Pi°>OoG,+<l-Ga)(W°-l]) (42) 

""^ PJ°H>«GWl-Ga)(P^Jn-lD (43) 

<?*c.«W=^^«?A,P/) yCkeG (27) PinH>A^a-GJPin-l] (44) 

65 

Pe[nJ-G«Ge+(l-GjPe[n-l] (45) 
PJn)-G„G.+<l-GjPin-l] (4^ 
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d»l of a» oil,, bud's oli«„. bSSL SSZ?^ 1^2."* *?" l«lb Pi lo .ri, uinSrSiiTT 

par^ete^ before idemi^ fcft hand coXte. '^HAiJ.}-«JnFHAUn}-«r„j,. 

no. 23 shows the preferred embodiment of ihe finoer r " '-'^•''> (52) 

fdenUfi«Uonprocess24«. Forthe contacts ^sZZto^ preferred because it speciaUy favors assignments wherein 

te^Iate of hand part attractor points, each attractor point ^'^ '^"'^'^ to those conS 

iTZr^^.'^'"^ '^'^ correspond to a particul^S ^r^?:?"*' '° biomechanical constraint thaS: 

Tbe^;!SS-~-^^^^^^^^ - atJSr4-?f---ach.nt^^ 

v^be M unfiUed. t.e.. assigned to the null path or dummy fe«ThelS''^rt'tt:? ^''±'7^''^ 

rrsssL" ^^^^^ 

^ons (Fi^r ) employed in hand oflfect estimation fi^,^^ * preferred embodiment, each weiehtinB 

^J/^-^'i-ia.traCor points tend to lie i„ a ring as f^fr^"' ^ ^" ^^^^^ "^^^^^^^^ 
displayed by the crosses in FIG 24 TTie Menriri^ ^ ^ uniquely resembles the given thumb ormlm hl!^ 

S.^ostlS^^r^'^' vahies^if ^tr^ftu^-J 
parts 201-207. If the given hand is a left hand the attractor « ^f^^ u- ^""^ attractor-s hand part The factor 

shown. FIG 24 also mcludes line segments 380 forming the iri^^ tab fcs, p.ecewise hnear fiinctions, polynomials 

VforoDo, ceU around each attractor pZt Every S^tWn '^•'""T?' ^"'^^'^ fiuK^ons/or^™ W 

an attractor's M,ronoi cell is closer to that att^c^rl^^" T°1 °^ '^e^'^nt betwee^a ^n^t ^ t 

-«>-ly onecontSrriJ^u^^^ « S^^^ - ^vored asTe^^t^ 

r^^s^^^rth^^rr^^^^^ s^.S;r^-'^--^--ofthe^^ 

multiple surface contacts in a band clSS^h^^°jd'^ on^^^^'r'''^ '^^ "8^. inner palm becl 
he u, the same Vbronoi cell, so the assigm;^m aSu^ « JS^ Z^'^' conta^Tfitt^ 

aU of the contact positions at once. ' ' fingetip and outer pahn heel cont^TZ 

usuaUy very close to vertical(90"),a^odentio"ofTel^ 
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where onemation is inconclusive of identic and re«:h«a • • • . 7} which mmunBcs: 

mimmum at 60«. ibe favored orientation of the opposite < 

thumb or pahn heel. TTk corresponding relationship the V 

fbout 90° ""^ orientation factor b flipped ^ 

FIG. 25B approximately plots the thumb size factor. Since . 

thumb si2e as indicated by total proximity tends to peak at „ • weighted distance from contact i to attractor 

two or three times the size of the typical curled fingertip the > attractor j arc considered assigned to one 

thumb size factor peaks at these sizes. Unlike pahn heels. Y "'"j" combinatorial optimization 

thumb contacts can't be much laiger than two or three timS ProWem, knowi more ^ificaUy in mathematics as an 

the default fingertip size, so the thumb factor drops back .t"' ^ 'fficienUy solved by a variety of 

down for larger sizes. Since any hand part can aSsmS n'^^T l?*'^^'"^^! techniques, such as branch and 

when touching the surface very liEbu7oriLSr , " combinatorial search, the Hungarian 

touchdown, s^all size L noT dfeS^fs^^tv ^«!.o^°etwo*flow«lve«.TT,oseskiUedinthe^wa^ 

factor defaults to 1 for JL l^fS^^' ""^ ^^f^ '^[^ type of combinatorial optimization prob- 

FIG. 25C approximately^prtslhT r heel size fac^r. ^^ll^^T^^^^^^fJ^J^^^^^ 
l^^J^ ".T'^ '^■'^ 20 -•-of'taalpa-meteltucTSS^rnT.^e'^^^^^ 
Z^^.f^^rt^Y^-'^'^^'^y'^'^'^^y P'^ identification method as the <Sof altra^? 
do so. Thus the palm heel size fector is much like Oie thumb contact distance minimization remains ^U^tWn ^e ^JL" 
size factor except the pahn factor is free to increase indefi- of this invention. ^ wen wiimn tne scope 
I^i w '^"'^^ can grow by becoming J° f °i<> ""wecessaiy compuution. decision diamond 360 
taller as the fingers are flattened. But ance finger widtii is „ ^'>&^' identification process at this stage if the hand 
constant, the eccentricity of an ellipse fitted to a growing fssigmnent of the given contact cluster is only a tentative 
hngertip contact increases in proportion to the height To nypo'hesis being evahiated by tl» hand identiflcatmn mod- 
prevent flattened fingers ftom having a large palm factor the c , " ^^^^^ ^ <bc given hand assignments are the 
size measure is modified to be the rati) of total contact Preferred hypothesis, fiirther processes verify finger 
proximity to contact eccentricity. This has little effect for "^™"es and oompfle klentity statistics sudi as finger 
palms, whose eccentricity remains near 1, but cancels the ^ "'^ ■a . « .- 

high proximities of flattened fingertips. Though diiecUv ^7?. '^.P.'?*"*^ ^ attractor assignment 

using fitted ellipse width would be accmSe for low Zt^,^ tehable when aU five fingers are touching 

resolution electrode airays. the above ratio^luy Ji^^tTwn P'^'° ^'''''^ '^''^ 

tures contact width. o oasicauy cap Checkmg that the honzonlal coordinates for identified 

Another important distinguishing feature of the o,lm S^'l^ ^""f.^ ■°^'»'"g easfly verifies that 

heels that ^ anato^fl^^ufrrtr^L'^f £ ^^^^^ ZrX^ZllS^^''7''"' 

Mveial centaneters. Hus is nrt tme of the thumb and assigmnent of the innennost fineer co^ m!v^J?! 

S^Watlr ^ ^^'^ contact Se^^b.rtlTcl' 

''«='*i°''di>«'ond3(S2 employs a thumb verificaSonpnS^ 
^measurcdbysearchongforthcnearestneighborcontoctof 3«» to take further mea4ements betw^n ST"^^ 
a given contact and measuring the distance to the neighbor. «""»ct and the other fingers. If these further mea 

As plotted apprtMrimately in FIG. 25D. the pahn separation ^^cnts strongly suggest the imiennost finger contac 
tactor quickly decreases as the separation between the ^. "'^?'*'y'**ron&'be thumb verification process changes the 
contact and its nearest neighbor falls below a few ^^IS™**"' °f "oncnnost finger contact. Once the finger 
centimeters, indicating that the given contact (and its nearest .^"Sn^ents are verified, step 364 compiles statistics atout 
neigMwr) are not pahn heels. UnUke the size and orientation J^sigoments within each hand such as the number of 

factors, which only become reliable as the weight of the r:^"'^^'*^'^''^'^^!''^ of touching finger identities 
handsfullycompressesthepahns,thepahnseparationfactor „ r '*? P'"^^ convenient summaries of identifi- 

is espeaally helpful in distingushii^ the pata heels from ° °^ modules. 

pairs of adjacent fingertips because it applies equally weU to ^*=P* thumb verification 

hght. small conucls. ^ ^ ' ° ""o^- TT>e first 400 is to compute several vekxrfty 

Once the thumb and palm weightings have been applied ^V^m' innennost contact 

to the distance matrix, step 358 finds the one-to-one a^ign- „ '^^^'^'^ « Anger relative to the other contacts identified 

ment between attractors and contacts which minimizes the ^ ** inter-path measurements presuppose 

sum of wetted distances between each attractor and it's * ordering, they could not have easily been 

assigned contact. For notational purposes, let a new mabix ^''^^ ^ attractor distance weightings because contact 

[c^J hoU the weighted distances: identities are not known until the attiactor distance minimi- 

aUon IS complete. For tiie factor deswqitions below let FI 

f "nm^^j^ if y= 1 (53) * « innennost finger contact. FN be the next innennost 

finger contact, and FO be the outennost finger contact 



Ai,P, ». if2*ys5 separation between thumb and'index^&iKriroften 

l,m^^^F,^^_^ if Uiger than the separations between fingertips K^a- 

'Hinf>pc^j».j„PU„j„Pi^j^j„) if 7=7 rations tend lo grow as the fingers are outstretched. TTiere- 

65 fore an inner separation factor inner_5eparation_Jact is 
defined as the ratio of the distance between the innennost 
and next innermost filler contacts to the average of the 
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distances between other adjacent fingertip contacts, avg_ 
sq>aratioa: 

outerseparanonfacta mid 1, ~ ^ ^ 5 

[ avgseparanon I 
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The factor is clipped to be greater than one ance an 
innermost separation less than the average can occur regard- 
less of whether thumb or index finger is the innermost 
touching filler. In case there are only two finger conUcts^ a 
default average separation of 2-3 cm is used. The factor 
tends to become larger than one if the innermost contact is 
actually the thumb but remains near one if the innermost 
contact is a fingertip, 

SiDce the thumb rarely moves further forward than the 
fingertips except when the fingers are curled into a fist, the 
angle between the innermost and next innermost finger 
contact can help indicate whether the innermost fingpr 
contact is the thumb. For the right hand the angle of the ^ 
vector from the thumb to the index finger is most often 60**, 
though it ranges to 0** as the thumb moves forward and to 
120° as the thumb adducts under the palm. This is reflected 
in the approximate plot of the inner angle factor in FIG. 32, 
which peaks at 60* and approaches 0 toward 0° and 120*. If ^ 
the innermost finger contact is actually an index fingertip, 
the measured angle between innermost and next innermost 
contact will likely be between 30* and minus 60*, producing 
a very small angle factor. 

The inner separation and angle factors are highly dis- ^ 
criminating of neutral thumb postures, but users often 
exceed the above dted separation and angle ranges when 
performing hand scaling or rotation gestures. For in^ance, 
during an anti-pinch gesture, the thumb may start pinched 
against the index or middle fingertip, but then the thumb and 
fingertip slide away from one another. This causes the inner 
separation &ctor to be relatively small at the start of the 
gesture. Similarly, the thumb-index angle can also exceed 
the range expected by the inner angle factor at the beginning 
or end of hand rotation gestures, wherein the fingers rotate ^ 
as if turning a screw. To compensate the inner separation and 
angle factors arc fiizzy OR'ed with expansion and rotation 
factors which are selective for synunetric finger scalings or 
rotations centered on a point between the thumb and fin- 
gertips. 

When defined by the following proximate equation, the 
expansion factor peaks as the innermost and outermost 
finger contacts ^e at approximately the same speed and in 
opposite directions, parallel to the vector between them* 

50 



cxpftosioafact**- 

(FO^D3-£(F45lJO(nD) 



(56) 



C3qiaiisioii__fi»ct:«max(0, e^»mion_jkct) (57) 

where Z(FI[nl, FO[n]) is the angle between the fingers: 



60 



Translational motions of both fingers in the same direction 
pixxiuce negative factor values which are clipped to zero by 
the max operation. Computing the geometric rather than 
arithmetic mean of the innermost and outermost ^>eeds aids 65 
selectivity by producing a large expansion factor only when 
speeds of both contacts arc high. 



The rotation factor must also be very selective. If the 
rotation factor was amply proportional to changes in the 
angle between innermost and outermost finger, it would 
erroneously grow in refuse to asymmetries in finger 
motion such as when the irmermost finger starts translating 
downward while the outermost contact is stationary. To be 
more selective, the roUtion factor must favor symmetric 
rotation about an imaginary pivot between the thumb and 
fingertips. The proximate rotation factor equation below 
I^aks as the innermost and outermost finger move in oppo- 
site directions, but in this case the contacts ^ould move 
perpendicular to the vector between them: 

m taHnnfitrfc — 

>ft'Wn><%«3S>8m(njn^/l(Fl[n]^ 
^n)-^(FI[nlFO(nD) (59) 

iotatioa_fect-max(0, rotatioa_fect) (60) 

Since motions which maximize this rotation factor are 
easy to perform between the opposable thumb and another 
finger but diflScult to perform between two fingertips, the 
rotation fiactor is a robust indicator of thumb presence. 

Finally, a fuzzy logic expression (step 402) combines 
these inter-<»ntact factors with the thumb feature factors for 
the irmermost and next innermost fingpr contacts. In a 
preferred embodiment, this fiizzy logic expression for the 
combined_thumb_Jact takes the form: 

combin«l_thuinb_fe«i-(iiiiKr_flqnnitioa_Jflc^^ 
paBsion_jBct+rotation_fka)x(F^^ 

(61) 

The feature fiactor ratios of this expression attempt to 
comparc the features of the irmemfiost contact to current 
features of the next innermost contact, which is already 
known to be a fingertip. If the irmermost contact is also a 
fingertip its features should be similar to the next innermost, 
causing the ratios to remain near one. However, thumb-like 
features on the irmermost contact will cause the ratios to be 
large. Therefore if the combined thumb factor exceeds a high 
threshold, diamond 404 decides the innermost finger contact 
is definitely a thumb. If decision diamond 412 determines 
the contact is not already assigned to the thumb attractor 
412, step 414 shifts the contact assignment inward on the 
attractor ring to the thumb attractor. Otherwise, if decision 
diamond 406 determines that the combined thumb factor is 
less than a low threshold, the irmermost contact is most 
definitely not the thumb. Therefore if decision diamond 408 
finds the contact assigned to the thumb attractor, step 410 
shifts the irmermost contact assignment and any adjacent 
finger contacts outward on the attractor ring to unfill the 
thumb attractor. If the combined_thumb_fact is between 
the high and low threshold or if the existing assignments 
agree with the threshold decisions, step 413 makes no 
assignment dianges. 

The hand contact features and interrelationships intro- 
duced here to aid identification can be measured and com- 
bined in various alternative ways yet remain well within the 
scope of the invention. In alternative embodiments of the 
multi-touch surface apparatus which include raised, touch- 
insensitive palm rests, pahn identification and its requisite 
attractors and factors may be eliminated. Geometrical 
parameters can be optimally adapted to measurements of 
individual user band size taken while the hand is flattened. 
However, the attrador-based identification method already 
tolerates variations in a single person's finger positions due 
to finger flexion and extension which are as great or greater 
than the variations in hand size across adult persons. There- 
fore adaptation of the thumb and palm size factors to a 
person's average finger and palm heel proximities is more 
important than ad^tation of attractor positions to individual 
filler lengths, which will only add marginal performance 
improvements. 
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rJ^ti^l^ fT^'** Of an alternative melhod for inoor- depend heavfly on which side of the board the cluster starts 

Sfr^^T ^^r^'*"'!^ ^^"'"y °f ''Wch recently lifted^ 

?n„ ll^ f I'TZ"' """"f -"^^y •'•^•Pfij- -^^-npte. if the right hand 

£n ^ ^ ^ ? ' lifts oft the next conUcts which appear in the middle will 

wilhm a ^n hand 430. step «2 constructs a two- most likely be fiom the right hand Suching back down 

JrrT^ t°"r^j"^T* ^'^ l^ft "riddle af d dispS ^e 

f^^'^h ,,!^ ^' " "^if right hand. n,e division l^tween left and right^es of Se 

Sr^h?l!^ '^K T°* opti^tion then surface should therefore be dynamic. slXg to^^e 

fin* the short^ graph cycle connectmg aU the contact lo right or left according to which hand was most recenUy nTa^ 

pajs^ pa^mg through «ch once 434. Since hand the middle. Sin«le hand ofeet estimates TmSy 

«,ntacts tend to he m a nng this shortest graph cycle wiU retain the last known hand positions after liftoff. sudTa 

or^Zlolr """"^ "^'"^"^ " ^ toplemeS^ by tying the positio-S^of 

Tlie next Step 438 IS to pick a contact at an extreme is hand positions 

!^'l!^,K or outermost and Hough cases remain in which the user can fool the hand 

SSn^ alo^nd '^'^'^'^^''^ ^) pahn identification system with sudden placements of a hand S 

(decision diamond 442). Uns can be done using contact unexpected locations, the user may actually wish to fool the 

:^^Ti^^r'^'^^°' '° systeminthesecase7Forexample!users JtSyo^haod 

utih^^l m the thumb verification process and the attractor 20 free to use the surface may intentionally place tte hand " 

^l^i^h . , TZ°^ ''.^ ^T"' ^ ^ °ntotheoppositehalfofttesurfacetoa,^thechord4u^- 

concludes that contarts above are most likely fingertips, and operations of the opposite hand, nierefoie when a Zd 

H S Th ™^ '^'^^ '^'^y tou^down weU into the op^^te hZf 

IS ^ermos^pali apahn heel, step 446 con- the surface, it can safely be given the opposite K identity 
^^^'^.'^^S^'T^y'^^^^^^^ibc 25 regardlessofitstrueidentit^ArchingVbesurfaceacrossthe 

S^T-^h^^ H ^ "^'^ ^"^^ "^""^ ^ middlecanalsodiscourageusersfiomsUdingaha^Se 

pa^TTie thurnb and palm tests are then repealed for the opposite side by causingTwkward forearm pronation shoiSd 

contacts adjacent m the nng to the mnemostunta any other users do so F snoma 

thumb or pahn contacts are fi.und. Once any thumb and FIG. 29 shows process details within the hand identifi- 

£rir^«rKl"'^"'''!?'^J'^°'^'".""'^« » cation module 247'DecisiondiamonK5?firdetSL 

V h- !^ '^^-^ ^ '^8 and whether the hand identification algorithm actually needs to 

S^Si^ '^T r'T- ■ be executed by checking whetheTaU path proxii^ties have 

tor i^S "^^^ *^ T "^^^^ "r** ■""^"i^ ^"^ili'y °f the identifications, hand 

fi, J^^^fi H- constramts on relative positions, the and finger identities need only be reevaluated whena new 
fi^ venflcatK.n fimctions for each contact may need to 35 hand part touches down or diimbiguating feati^of e^ 

S«S?^^'°'"Vi"*T'^'^^''r°^'^«""'^ mgcontactsbecomestrongerTlielontalsizean^on^^- 

rdative to other conta«s m the nng and relative to the lion features are unieliabk mitil the flesh ftilly compresses 

est,matedhandoflset.mattractortemplat^ against the surface a few dozen millis«»2 Z'^S 

l^^e^Zi , ""^r "'■'"^ "^-fo- decision diamond 450 ex^^ 
atoctor embodmiem more efcgantly moorporates expected 40 the hand identification algorithm for each proximity imi^e 

a^es bct««=en contacts and the estoated hand oflEBet into in which a new contact J^Kars and for J^entT-S 

S imity imagp. in which r^totalproximS^fa^y^ 

Hand Identification IS needed for multi-touch surfaces contacts continues to increase. For unages in which prox- 

which ^e large enough to accommodate both hands simul- imities of existing contacts have stabLsd and 

conUu^appear,prth continuation as performedbylS 

s^ jomed such that a hand can roam fieely across the tracking process 245 is sufficient to retain and extend Lp 

S^rn-f T mI'^-^u T?*^ "'"'^ °^ ''^> identifications computed from pre4^ 

hand rientification would be to assign hand identity to each images pi«;viouj, 

^^LTf^^ ^-l^uV. *\«""!f to-ched Should the hand identification algorithm be invoked for 
tZl^Ai°' "^l^ 1!^ i^^^^thc current image, the first step 453 to define and positi«i 

S f,^^'^' '"""^ ^ attractor templates. Tli«e sh^d be 

right hah^es some-^oflhe-hand's-contacts would end up basicaUy the same as the attractor templates (HG 24 ste^^ 

Sfore m?.^ "^k!!, Jl?\'^^ ^52) used in within-hand identification. ««pSttoiS 

^^h^Tw ^P"^""!^ """f «^ «^ i^to and right rings must now be utilized at on^. Tlie default 

.n T PT."^°^ ^ P'^«"«='" °f relative.to one another should cor- 

SSr^^intr'ol'^ll rf^" ^'^ '••^f^"" "and contact positiTiL 

^me iden ity. Once all surface contacts are initially shown in FIG. 20A. Each ring translates to foU^ the 

identified^the pam tractang module can reliably «tain estimated position of its hand, j^t like the slLy 

existmgidenhficauonsasahandslides&omonesideofthe tation regio^ foUow the ha^cb in FIG. ^Kl^^Zl 
^fZ^A- 1 - ■ « attractor points can safely be translated by their correspond- 

r.l!^l^K , ^ "'T^ onentations and the ing estimated fingpr ofl&ets. Therefore the final attkctor 

toderndSTof (AjJn],AjJn]) for the left hand L and right hand 

mdependent of cluster position which distmguish a tone H attractor rings aie^ 
cluster of right hand contacts fiom a chister of left hand 

contacts. If the thumb is lifted off the surface, a right hand 65 l^i">UuJnl4-LFj Jnl^uU^ (62) 

contact cluster appears nearly indistinguishable from a left , • r t , . r , 

hand cluster, hi this case cluster identification must still t^^J-Uw^^Kt^j-J-l^MU* (63) 
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rJ-JJ^U« (64) tor distances. Finally. Ihe weighted left and right haodcosls 

R«J»-RM»>WW>^RW (65) added together and scaled by the lecqjrocal of a hand 

BasicaUy the hand identification algorithm wiU comoare "^.f^*"^". ^'^^ °!'!^,^ '""^ «he partitioo. 

thecostofassigninga,ntac.stoatt,SS^n^?r^ 5 U.iS^°Sol^ hV™.^'"''''^'"'''^''^ 

the other, the cost depending on the sj of^S S^H^.r^^^^'^^''°T^''"'^'=^^'^'^''*^ 

distances between each^ontam and its f^.?^"^ *^ Pfft"*?"" «»« as the lowest and records the 

and compared. IT* SoS effic.^ wTto^J^.f^ Uct identificauon processso that the thumb verification and 

hypoth^ is to defiTatT^fTgirSr^^ . S^aS^^ ^ -'"'^ 

rr; rrS^^gT'S^at^rt^SsT £ hayr^t" r.T'^'^f « ^ -^^^^^ •^•^ 

horizontal coord4ites and establishi™ a vSl Lt!,^ .k^^ '5'!.''*^ ^"^ " ^^S^' ^'^ of 

halfway between each pSS^SS to^S^ST '"."^l'^.'""* down in the middle of the surfa«, 

nates. RGS 30A-C show exanmir^f ih^^^ ^ ?^ '^^'^ ^"^^ "g*"- Therefore when a hand 

tou,s475andTra^aS£.12Ty^S fr: '^'^S .down in the middle of the surface and 

fixed set of conucts. Each cont^^SSLSJ^l.! toward one side. U probably came fiom that side. A 

hypothesis. known ak^TSS^to^ aU ^Z^l t ^^^T' t"^'' ^'""^ proximately in FIG. 31A, 

tod«left476of theconto^'rSlt^hl a^S ^ "^ Phenomenon by slightly increasing in valu^ 

contacts to the right 477 of the ^^touTare fi«m ThTH M c ^ f ^^"^"^^ """^ "^^g ^ward the 

hand. Cbntouns a^LsTn^™ m Z ^ ^Tj^, ^ k*^"^" of the board, thus .fccreasing the 

of the surface to ha.Te^Sit^ th,t^tS^™ ba^ccostof Uie handTlefi„=torisa£iuK*ionof the avfrage 

thesurfeceare&omthes^eS^t?t^S,fc^r S T^'*^ of "^e 

esiz« more contacts on a given h^d to^^T^K; ^ T° '^^"^ ^""^ Wgh speeds do 
^ tta'r "LT"'^^'^ ^"^'^ be .used by ^ t^T^ZTn.lZT^-''"' "^"^^ 

hySS^fL'Srcon^rS'^orhr'h"™* f . -^-^^^^un.ton.n^Ztors^.^ui.ntify^^ 

oZl.p or ZJ^^ TtS^oftte opSe h^"^!'' '"^i' ^ "^'^^'^e ^iguous 

sidering that each b^7^cZ^^ jT^S^ °f ^ vertical position of the 

contact, this reduces theTuXTof^ MeX ^ 35 ^'^.''^T w """"and also gives 

tations to examine &«m thousands to at mosVaSfZh tf??^ 'T'^f ? °^ handedness. Tbt thumb tends to be 

fewer hypotheses to examine. ti?^dS of*^ p^^ ^^tnZ\ °:^^ ^"P^ 

lion can be much more sophisticated, and if h^J^^ r ^'^l' '^^''^P^- ^ 

computationally cosUy eo, ana 11 necessary, handedness factor, plotted approximately in FIG. 31B, takes 

The optimizatfon search kxjo follows Its on,l . . .^''g^ of this constraint by boosting the hand cost when 

determine whichoftl^to^di^SecSnSi^o^ "he contact identified as the outer«.ost fingertip is more than 

partition of two contact chist^^^^r^rS^ ^""P»«> '^"'''"eters lower than the next oute™ 

tions and anangement of contac^Stto e^htSr^" T assigmnent for aU 

satisfy kno^^tomical and^o^SS T Z'^^^.'^f^-.^ ^"S- Since this causes 

The optimization begins by fi^^ '^'^-'"^d identification algprithm to fit the contacts to 

divider such as ttefcZ^'^^iSet^^W^ the wrong attraCor ring, finger identities become reversed 

45<0 any contacts to S of tl^ «^^^THS /^at ^he supposedly lowered pinky is truly a lowered 

and the rest to the right hand Sta,^^k^.^ fi °^ "^"^ ^^^^^cly, limited confi- 

identificationalg^tSofSi l^wSi^^tst handedness fa^r. lHough the 
finger and palm' idecddes to Si^S j^SSSS so o'u^' 'b^/ " "ll^* " ''^ 

Decision diamond 360 avoids the compuutionale^r^f Zmb S ^ creatmg an ambiguity m which the 

thumb verification 3«8 and statisticsTatS aKftwf ^ fingertips of one hand have the same contact 

tentative assigmnent hyShST^ ' ' ^ anangement as the fingertips and outer palm heel of the 

Returning to FIG 29 steo 462 cnnmu,^ « ft. .1. opPostte hand. This ambiguity can cause the handedness 

partitioaTOs coS mTl to "Zfto^^ T « T ^ '"°°^y «ow for an accurately identified 

tenUdveIyidentified«n^TitSerSJ^r^!rS: c£.^l' ^""^^ ^ 0° 

and how well the partition meets bet:^^"^;:^^ ^Sjol 

r^rtiJIS ^"L'LTT^LuS'iSfitS Distingui^ing intact cluster is challenging because a 

contact to ^assigned a^a^^p^J ^ t St^^S «, ^^'.TJ^r "^"^ '"^'^ 

finger identificatir, incluS s£ irf ori»E fe^l^ oulstretohcd, with the pmky and thumb of the same^ 

factors for thumb a^d pZl^^T^^Z^ "P ^ cm However, the pahn can stretoh ve^r 

the basic template fittii^ cost foH^d^i^^^St . <=ompaxison. placmg usefiil constnunts on how far 

thenweightS^asawSl^with^^rt^fLeiSig ^n J te ^ir'T 

.^tswhichareund^emphasL^^-Tat^- 
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and forepaln, attractom. Hus us^Sy causes ^Z,^ T'^:^''''^'"''"^°'^''''^<^°'^^t^r 
assigned to the cluster's palm attractois to be s^S tZo^ ^ «"«°"« «ed not be 

across the surface wider than is plausible fiTt.^ S , '^^^'^ ^"^^ Additional contouis couJd weav^ 

contacts ton, a single hand. To puLh such ZSonT^ ^^t"^'^'' overlapping thumbs, or they coM t^ 
patocohes.onfactorquicklydn,^bek>wone fo^eoTiJe S^'ffY. '° ^"^'"^ <^ ^ated haS 

hand cluster m which the supposed palm conuS^e ^ ^^^Z f "^"^ '"""g^ one 

scattered over a region la^erTsL. Tte e^h ^HHV ''-P">-idedfor 

greatly i.«ease the hand's basic S FIG ^11^ '^' '^'^ """^^-^ P^tio^ contours 
31C shows the value of the pahn cohesion factor vers^ 1 P^r hypothesis to partition the surface 

horu^mal separation between pahn contacts. TTms tori^^ more than two portions. On a surface large enough for 

=^.r«id can be efficicnUy measured by fi«lir.g ™^ 1^°^ " be necessary to dlLnine K 

and mmunum horizontal coordinated of all fonta^^nT i^IlT ^'^'^ °f hypothesS 

fied as pahn heels or forepalms and taking the^^n^ ,c '"S,'*^'="'°'«««ouis, the hand identification moAde wox^ 
^tweenjhe maximum andL-nimum.'S^^^ riSTt^nWT '''' ^^^r 

I^H r k"""*"™." "™ ^"'ti'^'J separation Ut^'^'^ ""8 was centered on the 

and ^e honzonta and vertical factors are multSicativeS !^ff ' „ ,1.^** '"""'"^ °° ^ allow 

Rr"^,'° "-e final palm cohesion fac^r. ^ ^^l^,^"^- "^l '■*'«ifi<=a«ion module would 
no. 33 IS an approximate plot of the inter-hand separa- 20 . '^"f a'^^^^tor ring. 

Uon factor, nus fkctor increases the total costs ofpaS; ^^ctor-based finger identification system 248 tiU 

-n whrch the estimated or actual horizont^KS "''"'^ Ltac^S 

Uiumte from each hand approach or overbpTisi^j^irS HoTT^ ^■'^'^ configuration (FIG 1^ 

by findmg the minimum of the horizontal'^olfeL of S 12^1 "^^^ ^ dis&^g^b^ 

hand contacts with respect to their com«ponding deS 25 ™i ""angement within the pen grip from the 

fiflger p«,uo,s. Simflarly the maximum rfthetorSnS IdW "S"'"* ""^ hand co^nfigurati™ 

offiiete of Ore left hand contacts with respect to thViT^Sr S?;* ^■^'^P'?^P'^'«""«'l«o"terS^c^° 
^ng default finger positions is fo^ If thel^e^^ S L * "^"^"^ ^uch the^urS 

between U«se hand offset extremes is small enou^ to £^r. ^"g'nuls out ahead of them. He pen grip 
suggest the thumbs are overlapping the same SnS 30 ^^r. ""^.' ^"^^"^ " ^ rec^SS 
region of the surface while either t-Tuching the Z^- '° ^riScation process to det^ 

floatmg above it. the separation factor ^mesv^s^aS aT^^"^^'^"'- 

Such overlap corresponds to a negative thumb sepa^Si the'^n^" hlS"*'? ^ harxiwriting recognition via 
the plot. To encourage assigmnent of contacts which are fine. ^ hand oonfiguratwn is that the inner grippi,™ 
withm a couple centimeters of one anotheTto^la^ 3, sometimes the whole hand wiU be pid^^ 

cluster the separation factor gradually begirt to C s^! ^^^."^"^ ««^g^g Ange? arr^^ 

mg Witt, positive separations of a few centimeteHr IS Zl'^ '-■"Poranly d^ppear. Tlrerefore 
Tbc inter-hand separation factor is not applicablft^ pSi ' ^Tl '"'^ ^^^"^^ •» inba^Z 

Uors m which all surface contacts are asj^ed to tE emtt. ^^'"^ ^"^^ '° 'be preS 

hand and takes on the default value of oif in Zs^^ ^ Z^blL T'^^ "^"^"^ '^"'P"^^ fi''«=™^f 
Altemauve embodiments of this hand identificatiM^K, ^ti-^^n « ^ ^'='0'^ and by using the 

c«« can mdude additional constraint factors S^^.^^ ^''^^ f P^i«o°s i" measurement of £,g„ 
well withm the scope of this im^ention. For T^ZZ f^^^'°*7^^'^ "^"^ ^^g'^'^ are lifted off the ^- 

velocity coherence factor could be computed to W par! ^ilt^^T" "^^^ '^'^""^ P^<^ effective h^ 
UtioTK m which all fingers within a duster slide at Zr^ri 4S a^n^ ^^r*^"^ temporarily retain the unique juttij 
mately the same speed and direction, though ea^h cK^ f^8"'°'=°V^f"'= '•f^yi^ back toward the nZal S 
a whole has a different average spe^I and dirSon ^"*P,P«»/''"« » few seconds after liftoff: 

Sometimes irreversible deciaons made by the chori o,^ t If"^ ^ "^^^ ^he pen grip detection 
mouon recogni^r or typing recognizer on L baS ^^^^.J^T" ^ dete^Twheth?^ 

«asung hand identifications prevent late diangesTthe 50 SIh^'^'''^'^'""'^''^^"'""^'^ !^ °o«.dedsion 
^entifications of hand contacts even when new^^ ^^t^Jf^'^^^'^'^^'^^'^^ <>^<^^^ 

image information suggests existing identificatioSTe ^'"^y^Partsin step 4«7 only if^en grip 

^ng His might be the case for a chord sfidL whi^h Z^^T^'^ ^ '^'^^ "^^^ ^^^'-^ ^e ^ 
generates input events that can't be undone, yet weU into the ^n.7^I°' """acts. Hus ,he 

c^^mT 55 ^S?f.'jr'''°'^"^°*'"'^'''°'«^bandwr 

chord should have been attributed to the opposite ha^ h ^'-^^ ^'^P ^ -^trieves the 

^.scase^eusercanbewamedtostoptheSaind^k whfchS Tr"" ^f' ^'^'"^ ^ Pal"- heels 

for po^ible input errors but in the .Meantime it is b^Tto '^''''f 'he surface. 

retain the existing identifications, even if wn,ng ratheTtha^ ^ «9 computes a knuckle factor from the outer finper 

switeh to correct assignments which couldla^lX? ^ ^ "^^^ fiom the palm heels S 

unpredictable effects when added to th^ eioo«us^ * 

events. Therefore once a diord slide has gener^S S S^T f"^/*"* to the pahn heels. Step 490 compul« a 
I^V of their existing%aS mayT ^^h '^^ *!f verticKr! 

locked so the hand identificaUon algorithm^ only l^a^ fnHel « ■"""I '^'^ ^'^'''^ '^i^^b Peaks as the 
Klenufications of subsequent new contacts. ''"'^ ^''P ^'^^'bp juts further out in front of the knu^^^ 

This hand ulentification process can be modified for t^Z^^^''^'"^^°^^"''^^^'^'^^'"^f^i<Jc 
diflerenUy configured multi-toud, surfaces and rema^^fl TZ^^ P'*^°'« ^«=^'*^a 

am weU an autoregressive or movmg average filter. Decision dia- 
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disabled for the pen gripping hand. , Whc mulUple degrees of fi^on. cL^ 

to pen grip mode, decision diamond 493 determines .h^^ same tune they are said to be integral rather 

whether Ibe mner gripping fir«e« are actually touSX ^aS"^ T^'^ ^ fasLr 

surface. Kso, step 495 generates inking events fiom^ oath '4'°'' ^SPn^ -"otions rather than reirictiS 

parameters of the inner fingen. and Ippen^rTto'^^ ^"'^'"'^^"-g*'^ "isordegreeof fieedom atS 

outgoing event queue of the host communication interface ,n Wh, , r 

TTi«eutogeventscaneithercause«digi,ali,^tobLS^^^ ^ T^" °°'yf°"''^8^°ffi«=«tom are needed, the basic 

onthedu;play24ford,awi„gorsignat.^^Sn=pJS S"lv7 '^'^ hand^gt^ 

tl^^'^^ ^ h'^d^rii Sr^K^^°"*^'^'^'^g^'^'^»''«'d.otrtion 

system and mten,reted as gestures or languLe te^L^ fi'^'' unscrewing a jar lid or 

Himd«^Ungr«x^ition systems are weUto^^i^X^^ ^S JSlLd'^?'' '^^^ '^'^S °"«- N<>' °'>'y 

If the inner fingers are lifted. Step 494 sends stylus ^•T^^'^.^'^y P**™ f««ause they utilize 

hl^^^: -""""nicatirinterface to TrS^^ ll^v 'T''^ "h-^b!^ 

handwntingaxognitionsystemofabreakbetweensymbok SS^rT"^!'''. '? mampulation taste 

InsomeappUcaUoostheusermayneed toindicatewKhe S,^' f""!'^''?"^ Tl^ir only drawback is that the 

di^lal ink" or mteipreted symbols are to be inserted on Se 2n of aU the fingers during these han^ 

di^lay by positioning a cursor. Tl»„gh onTS t^'Jh 2^°^^ '° '^'^^ P^rf^X ^ 

sur&ceausercouldmove±ecu.sorbjHbavingthepln^n ^ * translation in some direction T 

configuration and sliding a finger chorf. it SSf^o *^ °' To aUot ^ 

aUow cursor positioning withom leavij the ^ n SJ^i Jfr*^ ='™"J'>^"sly so Lt the d^S 

figuration. This «m be supported by^ene^TlH.^ 25 P^^"" •«i"'«=»ded trS 

positiomng events fi«m slides of the ^ heels mnHn^^^**^? «=»^8S and rotations, the 

touckles. Smce normal writing motioVwill aLTLCe "°''?"."*«^«°r preferentially weights the fingers ^ 

shdesof the palm heels and outer knucklSpi^mTtitS ^n^'ZTT''^^'^'^^'y^^y^<^t^- 

should be ^red until the inner fingers ha^ Lu Iffl^ f^ -tepcndrng on their speeds relative to one a^the" 

a few hundred milliseconds. oeenunedtor ^« P'<>«sses within the motion component extrartor^^^ 

Should the user acmally pick up a conductive styh« and ^ ^ ^ ^^^^ uS^L 

attempt to write with it. the hand configuration v^^J^a^e T'^^^.^^^'^^eivcnbaj.d. These paO^con^Z 

s^Uy bemuse .be inner gripping fin^ wS?XSg M^"","^ 'T^'''' to be used in the Ttion 

ae Stylus from above the snrfiice rather than touchiSttf -dentifications are needed so that 

surface during strokes. Since the foreann tends to Sie 35 "T^^ '^-^^'^'^^P'^'"*'^^!' would degrade 

^^aS^a;srsiLi?rh'^^^-''^^' '^"^ 

of the pinky ring finger and ihe outer pZ ES^^Ste con^'.°*? SCttaPPMes additional filtering to the lateral 

outer pahn heel may lie further outw^ than JZT^b ^^^'''''^'''^'^"^"g'^P^'^n-'y isch^gingraS 

respect to the pinky, the ring and pinky fingered ^ ^ I^"' ''"^g ^» liftoff and toS 

appear as large knudde contacts ci^ctose to tTouS ^""J^ "^f' P'^ ^he fingertip oS 

pahn. The hp of the styhis essentially takes the pla« of t^e lifts off after the tack of ^e 

mdex fingertip for identification pu^oses. remS^^t i.' ^""7 " °' "P^"^ laterll tfal 

above the vertical level of the knuddTrhus ^n Jv S^^,. ProrimityKlependent 

detector can function in essentially the same v,^^n^ 45 ^^l^ P»t to good use when slowly Xg toe 

sent to 4e host communication interface will in altuXv be ^fj^^.-^^ T"^ ^ ^ j-^P away from a 

caused by the slyhis. ^ Position during finger liftofiF. This prevented b^ 

Technically, each hand has 24 degrees of freedom of '^'^ !^ * ^g"''' ^locity in pr.^ 
movement m all finger joints combin^but >sVT^ti^ so o^fi '^'g'' changes in the finger's pioximij. Sk,^ 

matter, tendon linkage limitations mate it ^^^^ oS^.'^'^ri'^'' '''^ '° ^'^''^ fi^« 

all of the jomts independenUy. Measurements of E ^.fn ^""^ «J°J^««li»g of each finger vetodhfis 

contactson a surface yieU ten degrees of freedom to mSn '° "^'^ P«««°' in P^x 

^terd to the surface, five degre^fo^^i^iSS « ^^"'^ Altematively, more pS 

j8«^pressireorproximitytoti«surface.andoneX^ S^^d"" ^ ^"^"^ "''''™'=^g 1^ 

HrJ^r orientation. However, many of tb^ fi P^P^rti^-^J ^ the instantaneous 

degrees of freedom have limited ranges and woli remi^ . '^'g^'" '^^ '^«» that the 

unreasonabletwistinganddexterityfilmtheavei^eZto "/'^^^ ^^g^^'^'^i'y --«=d by finger lif,^ 

acc«smdependently. ge user to ^ Pn>I»rt.onal to the change in contact hei^t ^e to the 

16klfT!:°/'^''r'**"'«"°P°°«°'«^«=»iot»tnodule 60 ^^^''s^^ ^ °' '^'^ '^^ ^ 

ellh ''•^ of freedom mea^ ''"f^"' '^P' ^ 36. 

enough degrees of freedom for common graphical manin^ Slf I P**'" '""'"^"J' components from radi J 
lation tasks m two and three dimensions. iVtWo XnS^ fcf^l" « «=««ct^ 

rotating, and zoommg or resizing, in UueedimensioTu^o 6S Se h^HT^"* *"*'^°'»»g^velocS 
additional rotational degrees of freedom are av^We J ,t „r^ f?"'* '^Z™'' ""'^^^ i« ""d" '"ore 
around the horizontal and vertical axes. Tbe motion^S r^L^Zt^frt'^''''''^^''^'^^'^ 

roiaiions are eaaer for tiie user to perform if one of these 
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fingeisisdieopposablelhimib butthem the thumb verificatioD process are only sensitive to sym- 

Sr°^ that le^ than two fingers are touching the fingers. Ihey approach z^ro if any signifiLt whole haS 

surface, step 524 sets the radial and rotaUonal velocities of tran^latinn nrrl^rHno th^ wooic nana 

the hand to zero. no. 35 shows trajectories of each fineer 5 ^"^^^^""^ """"^ 

duringacontractivehandscalin^Tl^ethl"2W^ ' menta^Jn c- the 

205 travel in nearly opposite di^ons at roughly the same "^"^ "^Tl °° °' ^"^^ 

speed, so that the sum of their motions cancZfor zero net equations of the approximate form of Equations 56-^ 

translation, but the difference in their motions is maximized ^ «jmpute rotation and scaling velocities between the 

for a large net scaling. The central fingers 202-204 also lo touching fingers other than the outermost, 

move toward a central point but the pahn heels remain resulting velocities are preferably combined with the 

sUtionary, Ming to complement the flexing of the central ^sults of Equations 66-68 via a maximum operation rather 

fingers. Therefore the difference between motion of a central average in case traaslational motion causes the fixed 

finger and any other finger is usually less than the difference P^"*^ rotations or scalings to be zero. Finally, decision 

between the pinky and thumb motions, and the sum of is diamond 532 orders a check for radial or rotational deccl- 

central finger velocities during a hand scaling adds up to a eration 534 dutring motions prior to finger liftoff. The 

net vertical translation. Similar phenomena occur during method for delecting radial or rotational deceleration is Ihe 

hand rotations, except that if the rotation is centered at the same as that detailed in the description of translation extrac- 

wrist with forearm fixed rather than centered at the tion. 

S^lt^^i'LTi^^^ - HG. 37 Shows the details of hand translationalvel^^^ 

Since the differeiL in finger mol^^A^Uy greatest Z'^T . ""^'^ 1 
between thumb and pinky, step 526 only retrieves^'^^ Tin T ^''"P^^^^ ^and translaUon velocity would 
and previous positions of L imiermost and ou^^t ^ ^ '^P?^ ^"'"^^ 

touchii^ fingers for the hand scaling and roUtion measure- 25 ^'^^''f ' ^"^^ ^0^°° control to display 

ments. 8^ ^ be constant regardless of how many fingers are being 

Step 528 then computes the hand scaling velocity moved, even if some are resting stationary. Furthermore, if 

from the change in distance between the innermost ringer FI ^ simultaneously scaling or rotating the hand, a 

and outermost finger FO with approximately the following simple average is sensitive to spurious net translations 
equation: 3q caused by uncanceled central finder motions. 

Therefore, in a preferred embodiment the translational 
^ (f/[n]. FO[n]) - diFiin - 1], FO[n - 1]) (66) component extractor carefully assigns weightings for eadi 

^ fiiJger before computing the average translation. Step 540 

initializes the translation weighting Fi^ of each finger lo its 
where d(FI[n]J?qn]) is the squared Euclidean distance contact proximity, i.e. Fi^nH^iJn]. This ensures that 

between the fingers: fingers not touching the surface do not dilute the average 

with their zero velocities and that fingers ^ch only touch 
d(Fi[oy^nl>-V(FUaj-FOinI)2+{Fyn)-FD^nI)=' (67) Ughtly have less influence since their position and velocity 

to'^h^^^tiT™^ - - °- ^ ^^^g^Tf s^^^ 

putes the hand rotational velocity from 1^ ^^in T 7f T ^^^^^^ ^^ng 

angle between the imiermost and outermost finger with '^rdmg its sp^ 

approximately the following equation: 45 T^^^ ^ eadi finger's translation weighting 

in proportion to its speed divided by the maximum of the 
H^[n] = (68) ^eeds, as shown approximately in the formula below: 

50 \^°^jf^fjpttd[n]) 

The change in angle is multiplied by the current separa- 
tion to convert it to the same units as the translation and ^^^^ ^ power ptw adjusts the strength of the ^)eed 
scaling components. These equations c^ture any rotation dependence. Note that step 544 can be skipped for applica- 
and scaling components of hand motion even if the hand is 55 ^ computer-aided-deagn in whidi users desire 

also translating as a whole, thus making the rotation and ^ normal cursor motion gain mode and a low gain 

scaling degrees of freedom integral with translation. mode. Lower cursor motion gain is useful for fine, short 

Another reason the compulations above are restricted to range positioning, and would be accessed by moving only 
the thumb and pinky or innermost and outermost fingers is one or two fingers while keeping the rest stationary 
that users may want to make fine translating manipuktions 60 Step 546 decreases the translation weightings for the 
with the central fingers, i.e. mdex, middle, and ring, while central fingers during hand scalings and rot^o^though it 
the thumb and pmky remam staUonary. If changes in dis- doesn't prevent the central fingers from making fine t«ns- 
tanccs or angles between the central fingers and the thumb lational manipulations while the thumb and pinky are sta- 
were averaged with Equations 66-68, this would not be tionary. The formulas below accomplish this seamlessly by 
possible because central finger translations would cause the 65 downscaling the central translation weightings as the inae- 
appearance of rotaUon or scahng with re^ct to the station- nitudes of the rotation and scaling velociti^become si^- 
ary thumb or pinky. However, Equations 56-60 apphed in nificant compared to K^^,^- 



us 6323,846 Bl 
45 46 

thai the decision after liftoff to contioue cursor motion 
Fi^[n] ^ pdi!lil££^=^ (70) depeods on the state of the deceleration flag before liftoff 

K^^4-\HAn]i began, 'ilie final step 560 updates the autoregressive or 

rUn]xK,„^^ (71) moving window average of the hand translation velocity 

^ ^ + + \HM ^ vector, whidi can become the velocity of continued cursor 

motion after liftoff. Actual generation of the continued 

cursor motion agnals occurs in the chord motion recognizer 
where these equations are applied only to the c^tral fingers 18 as will be discussed with FIG, 40. 

whose identities i are between the innennost and outermost Note that this cursor motion continuation method has 

Note that smceha^^^^ ,0 several advantages over motion continuation methods in 

nn? f^''^ ! honzontal translation weighting Fi_[n] rented art. Since the decision to continue motion depends on 

^^rT^^ff^ ^ ^ ^ percentage acceleration which inherenUy no^E t^ 

mdjcated by the lack of a hand scahng term in Equation 70 anv ^j^A^t^cr^ tu^ - * "^^'""""^ uurmduzcs lo 

The translation weightings of the innermost ai^oute^^ any spe<^ ra^e. the i^r can mtentionally mvoke motion 

fingers are unchanglS^fthepolar^^^^ ~aUon from a wide range of speeds 

n^TnVR fnVFI fnl and FO fnUFO nVL^n ^P^" ^"^y stow motion 

[n]^teJ^M8^^^^^ Sr'?/' n^r" a document at readable speeds, 

vector (UMM^nJ) the weighted average of^e I^^^f Jh^ll^^'^f "^^'^ m U.S. Pat. No. 
finger velocities- 4,734,685, which only continues moUon when the user's 

motion exceeds a high speed threshold, nor of Logan et al.'s 
J 20 method in U^. Pat No. 5327,161, which if enabled for low 

YiFi^Fi„ filler speeds will undesirably continue motion when a user 

/,^[„] = decelerates on approach to a large target but fails to stop 

^ . completely before lifting oflf. Percentage acceleration also 

^j*^^ captures user intent more clearly than position of a finger in 

25 a border area. Poation of a finger in a border area as used 
5 (73) U.S. PaL No. 5,543,591 to GiUespie et al. is ambiguous 

2j '"'-r'^r because the cursor can reach its desired target on the display 

ff-yW = j"ft as the finger enters the border, yet the touchpad device 

V will continue cursor motion past the target because it thinks 

ti 30 the finger has run out of space to move. In the present 

invention, on the other hand, the acceleration ratio will 
Ti,,i,rf„.w f.K ^ ... ... . remain near one if the fingers can slide off the edge of the 

I,., ?^ fT^f""'"'!?^*^""*'"'*^^"*'' '"^y bitti^ » physical barrierTsensibly 

lateral deceleraton of the fingers before liftoff which reli- invoking motion contimiation Bilt if the fingers decelerate 

,y^l:^ . °' »° ^ edge of the se^g array the 

at liftoff. If deceleration IS not detected prior to liftofl^ the cursor will stop as desired 

user may mtend a special "one-shof command to be process 508 are shown in FIG. 38. Fingertip proximity 

Uon tests while finger proximiUes are not dropping too 40 tip normal to a hard surface. Unless the surf^ itself is 
qmddy to prevent the perturbations in finger centroids highly compliant, the best dynamic range of fingertip pres- 
whichcanaccompanyfingerliaoff&ommterferingwiththe sure is obtained with the fingers oitc^and hand 
deceleration measurements Step 551 computes the percent- nearly flattened so tha the compressible soft pulp under- 
M hT™", T° °^ ^ KH« "eath the fingertips rests on the ^ace. Decision diamond 

WM^aJH to a past average translation ^leed preferably 45 562 therefore causes the tilt and roU hand pressure compo- 
^r^^. " movmg ,^w aven^ or autoregressive nenis to be set to zero in step 564 and pressure extraction to 
filter. I^K>n diamond SS causes the translation decel- abort unless the hand is nearly flattened. Inherent in the test 
Tth^^ ^-Ik J^J^" is than fi>r hand flattening 562 is a finger count to ensure that most 

a threshold. If this threshold is set greater than one. the user of the five fingers and both pahn heels are touching the 
will have to be accelerating the fingeisjust prior to liftoff for so surface to maximize the precision of the hand priSaire 
^LTr r ""T'i 'lf li'^'^" ^ ^' measurements, though technically only three nonnSjUinear 

cureor moUon wJl rebably be contmued as long as the hand Contacts arranged like a tripod are necessary to esUb- 
^r mamtains a coi^t lateral speed prior to liftoff, but if lish tilt and roll pressures. Deci^on diamoS can al«) 
i^^i .1 T , '^'o " ^^^^ " the user to explicitly enable IhreeKlimemiional 

area of die AsplayUie deceleration flag will be set. Decision 55 manipulation with an intuitive gesture such as placing all 
dumond 554 can also cause the decelerat«,n flag to be set if five fingen; on the surface, briefly tapping the palm heeb on 
Uie current translation duecUon s substantially different the surface, and tinally resting the pato htels on the surface 
from an average o f past directions. Such change in direction Decision diamond 566 caus«^ step 568 to capture and store 
mdicates the hand motion trajectory is curving, in which reference proximities for each contact path when the prax- 
es cursor motion should not be continued after liftoff 60 imity of all contacts have stabilized at the end of this 
i^^A determmmg ae direction to the user's initiation sequence. TTie tilt and roU pressure components are 

intended target becomes very difScult. If neither decelera- again zeroed 564 for the sensor array scan cyde durinu 
Uon nor curved trajectones are detected, step 558 cleats the which this calibration is performed 
traiKlaUon deceleratiM. flag TOs wiU enable cursor motion However, during subsequem scan cycles the user can tilt 
n1 Tf H "^^^ fingers subsequenUy begin liftoff. « the hand forward appi;^more press4 to the fingertips or 
Note that decisjon diamond 550 prevents the state of the backward applying more pressu^ to the pahn heel. oVthe 
translauon deceleration flags fiom changing during liftoff so user can roU the hand outwaid onto the pinky and outer pahn 
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heel or inward applying more pressure to the thumb, index 

finger and inner palm heel. Step 570 wfll pnxeed to calcu- -continued 

late an unweighted average of the current contact positions. //„W: = /M„jxf-^2:^21_r ^ 

Step 572 computes for each hand part still touching the ydcminonapted) 

surface the ratio of cuirent proximity to the reference S 

prorimity previously stoied. To make these ratios l«s where H^^J[nl H^^n], and H^Jn] arc autoiegres- 

sensitive to acctdenul lifting of band parts, step 574 dips sive averages over ^e of the traction speed. S 

them to be greater or equal to one so only increases in speed, and roUtional speed, wheie: ^ 

proximity and pressure register in the tilt and roU measure- , . . r , 

ments-Anotheraveragecontactpathpositioniscomputedin '° '^■°^'-^P-'-™-(H^.l ,t_j.l 

step 576. but this one is weighted by the above computed controls the strei^th of the fiher. As pds is 

proxmnty ratios for each path. The difference between these adjusted towards infinity the dominant component is picked 

weighted and unwei^ted contact position averages taken in out and all components less than the dominant tend toward 

step 578 produces a vector whose direction can indicate the Producing the orthogonal cursor effect well-known in 

direction of roU or tilt and whose magnitude can control the " drawing appUcations. As pds is adjusted towards zero the 

rale of roU or tilt about x and y axes. fillers have no effect Preferably, pds is set in between so that 

Since the weighted and unweighted position averages are """"ponents significandy slower than the dominant are 

only influenced by positions of currenUy contacting fingers f\uAer, but components close to the dominant in 

and mcreases in contact pressure or proximity, the method is 5* ^ ''"^'^ affected, preserving the posability of 
insensitive to finger liftoffs. Computation of reference- ^ ""8°"*! '"°''<"'">n"iltiple degrees offreedom at once. The 

normalized proximity ratios in step 572 rather than absolute autoregressive averaging helps to pick out the component or 

changes in proximity prevents the large palm heel contacts «"nponents which are dominant over the tong term and 

from having undue influence on the weighted average Dosi- siqjpiess the others even while the dominant components are 

tion. slowing to a stop. 

Since only the current contact positions are used in the " ^"^J"^ * 

average position computations, the roll and till vector is i filter. A dead-zone filterproduces zero output 

independent of lateral motions such as hand transitu o^ !^ velociUes less Uian a speed threshold but 

rotation as long as the lateral motions don't disturi> fineer SZ^.u**''?^"* ^ proportion to the difference 

pressure, thus once again achieving integrality Ho^wr « Tu' ^P^J°<' '^"•d inP"t veloci- 

hand scaling and differential hand JressuT^ dSH^' * ^«'^'««fd'h«=^«hreshold. Prefer 
use at the same time because flexing the fingeis eeneraflv I is set to a fraction of the maximum 

causes significant decreases in fingertip c^act fiuZT^'-T"""" T^" ^ components are 

thus interferes with inference of finiertta pres^^a« filtered usmg tte same dead zone width. The final extracted 
Whenthisbecomesaseriousproblem,at^d^SS « component vetocit,^ are forwarded to the chord motion 
component can be usedasas^degr;eoffeSi^^ ™T^' ""f^f. ^ ^ 

of the hand scaling component. Hif^tal p^ ^ ""^molt t:^^ f^^'i 1^^ '"'^'^ 
nent causes cursor vekxrity atong a z-axis in proportion to ^ . ? ' shows the detads of the finger synchronization 
deviations of the average of thf con^^i,Sy «^ H ^^J"^,"^' ^ synchronization detection process 
from one. Alternative embodiments may include further ^ s^^f^ "^^^ ""^^ independently, 
enhancements such as adapting the reference proximW«to fh^K^. V^oiamAy markers and idenUfications for 

slow variations in resting hand pressure and^iTjZ^TSid ^LT" " P^^hs- J^^ identifications will be neces- 

-'-^'^ p"^;iSSd'=-;rriKhS'e:^s.^t^s 

T^Zt^^^i^T^ components m such drops below a release proximity threshold prior to totS 
TSsioZln 7Sr ^"'^ "l'^ "P?"^ Step603searchesforsubsetsof fingetswhichtouchdown 

J,^^ ^ P™^^ ^^i- of the palms is to su^rt the forea;^ 

whfle the hands are resting, palm activity is ignored by the 

^[Fng 12 and chord motion recognizers 18 except during 
w„[rt]: = //„[rt]x( -~2=fl!^p dififercnUal hand pressure extraction, and pahn heel presses 

K^nunaruspeed) 60 can be cxchided from this list and most other synchroniza- 

H r , « r . r V** (75) ^^^^ ""^^"^ ^^"^ synchronization between the two 

recent entries m the hst are compared If the difference 
= ",M-f "'^^"^ V between their press times is less than a temporal threshold, 

(dominaiaspeed) ^ "1® lincr presses are considered synchronized. If not, the 

most recent finger press is considered asynchronous. Syn- 
chronization among three or more fingers up to five is found 
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IS within a temporal threshold of the nth mos. S ^Z^l^ T^^'^ ^'^ ^^'^ "^"""^ if 

synchrooizaUon among the n most recent fingeT^^l' SSI'^ThT.^"'^ * '""^ ''^""^ '"^ aborts 

mdicated. To accommodate imprecision in tonV+^J! . '''^''yP<"'»«f that the presses were intended as key taps, the 

acr«« the hand, the magnitude ofT,:^"f'S^^^ ' - ^ •fl^ely to be key taps if the fin^ n.^^^ 

should increase slightly in proportion toT U^Z^n^T^' 

fingers being tested for synchronization. n,e largest set of moJ^iT^ ^ 

recentfingerpressesfoundtobesynchronizedis^rdrfL • leaves the synchronization marker pending so 
|he synchronized subset, and thTcomSi^n^ 4" lo 

rfenuues comprising this subset is stored convenienUri a fi^^ J ""^ next scan cycle. If the synchronized 

finger identity bitfieU. Tbc term subset is^te^^e ^ '^"'"^ firrfs with the 

synchronized press subset may not inchide all fi^^c^ syDchromzaUon release search «03 that they are 

renUy touching the surface, ^« ha^^ whe7 aln^; f""^ '^y°«='«'°<>'«>y ««=P «22 releases any holds on 

touchesdownmuchearlierthantheoZrfingersyetreS ,s Z^J^^'Ta'^'^Z'^'^^ '^^ "'^^''^^^ 

touchingastheysimultaneouslytouchdow^^oS^ ZhT *° ^ keypresses. Tlough the s^ncL- 

of finger release times is similily maintTedtTs^^^ at^ ^^.^ ".^^ '^J^' ""^yprei queue 

separately. Alternative embodiments rm^ reZe SI P^'^y^^y ^ave already been deleted in sti^ 612 

finger stiU be touching the surface to be'incIudS iS L i^l^^^l^'^'^.'^"''^'^r' '^'^r>624 

synchroDiZBd press subset synchronization marker, indicating that 

Decision diamond 602 checks whether a synchronization °°JT.'1*' synchro^tion tests need be done for this subset, 

marirer is pending from a previous image ch.^TTf '° ^ ^"^^'^ synchronized 

decision diamond 604 chedcs whetherle^^S fo^^ ^iT^r,^'^'^'^^^^''"^ 618 removes 

a newly synchronized press subset in the current proS ^^^ ^-^.'^V '"Us from the keypress queue in case they 

miage. If so, step 606 sets the temporal syn^hSS 25 ^ulT, " ^J!^'''^ "^'"^ synchronization test 

marker to the oldest press within the ne^^nSZ^ fi ? tT'^ ^'^'"^^ ^"^^ «'«^ synchronized 

^t. Additional finger presses may be aJL to STs^S S^. .k'^^'T' ^ '^'^ single 

during fumre scan cycles without affecting^ v^^e rf^L ^J'^^X^^Tf. "^"^ 

temporal synchronization marker. If there is curr^U v m ^ ""^^ » if they are to qualify 

finger press synchronization, decision di^onS^Ster- 30 ci^f^^ '^^^'^ 

mmes whether three or more fingers have just llnrelSS • xtm^dmn^ 

simultaneously. Simultaneous rdease of th^^ morel^ "l?*^ ^ '""^"^ P'^ synchronization 

gers shouU not occur while typing with a set of fiZ^^t fi ^^i' " '^'^d of 

does occur when lifting fin^ off Uie fi ^^"1 ""T"' '""'^ "'^^ ^ of Peak 
Tlerefore simultaneous relfL of thLT^rel^ 3S ^'.^^'^^^'^'^-^^^^^-^^o^Zmd^^. 

reliablyindicatesthatthereleasedfingersar^noT^teS^ " "^^^ "^""J lateral 

keypresses and should be deleted frim the kTjwSaSu^ ZlT ?f '»«^'. 624 clears the synchro- 

605. regardlessof whether these same fingers toSC ZLT ""^ "'"dusion that the syncbionized 

synchronouslyReleasesynchronization^f t^fi^et?^^ " '""^"^ ^"S'" or part of a chorf 

byibetfareliableindicatoroftypingintentandh^noeffect « If the choni t,n ^™^v 
onthekeypressqueue.Thekeypressqueueisdescribed ^^Tr . ■ Tk ^ tap conditions are met, step 630 looks up. 
with FIGS. 42-43B. q«:«>euidescnbed later "smg tiie synchronized subset bitfieU. any input events sud, 
Once a press syinchronization marker for tiie hand is ^J^°"^ ''"''^^ ""^ keyboard commands assigned to tiie 
pending, fiirjher pro«ssing checks the number ofXe" "fi**""! "^P- ^™ 
presses which are synchronized and waits for release 45 res&^hort^^ ^^^^ ^ "^^«^ ^ 
synchromzed fingers. If decision diamond 608 finds thm:^ fi,^h, v k ' ^ wiU 
more fingers in the synchronized press subs«. 1tet«Sn°[ S^,^^ 5^"'"; °° "^"^ 'f 
possibly be typing with these fiigers. nierefo^To^ ^ '^'P *PP*°* ^ main 
^.mediately deletes the three or mfre sySro^' p^ 2^ St'^^n'^S' ?^ communication interface 
from the keypress queue. This way they camwTaiEr^ sn 1^ ^ f 624 clears the synchronization marker in 
symbol transmission to the host, L tJ^^^ £ Tan^ synchronizations on the given 
symbols from subsequent asynchronous presses is nrt T.' , fi^^h 

blocked waiting for the synchronized fingers to be released rhZx , *^%.P««a'^° accidental generation of 

However, when the synchromzation^nly ^^Sl^ ^I'TL^^ '^""^ it is also usefiil for decision 

finger presses 608, it is difficult to know wLZ ss wh.Vhl "^"^ "''P «4 the first chord tap 

mtended to tap a finger pair chorf or intend^ t^ t^Z sWe^n^w """n'^Tf ' r'^''' '^yP'^ " ^ho"^ 

adjacent keys and accidentally let the key press^c^ l±=v I """^"^ '"''"S ^ ^ ^ 

simultaneously. Since such accidental simuLn^p,^ Z^^l K^L"'^.."^^ ^^""^ '^'^^S, requiring an 

are usually followed by asynchronous releases oTth^Z 1^^,77^ "^f"^ l"'"" " h^PPe-S to 

fingers, but finger pair chords are u^y relea^d 60 " "^ "^'^ ^ "^^ "''^^ '"^P 

synchronously, the decision whether the press« a^n ^ J? l"^"* """^ '^"c'' b"' ^is 

chronous key taps or chord Ups must be delayed until finder ">an having an accidental chord tap cause an 

release can be checked f^r syncTronSo'ln X ^™ "'''^ ^ "^"^ > 

meantmie, step 610 places a hoU on the keypress queue to dAA ^ .u ^ . . 

prevent transmission of key symbols from tte ^ ■ *T,o' '^^^ °^ '^"^ ""olion 
finger chord or any subsequent fi^gerpSL To Si! ^^Z'f^. '^'^'^ Pro- 
long backups in key tiansmission.'d«S^amoSd 6?4 STuv^J^'T " i«»^- 

oujuuu denUy. Step 650 retncvcs the parameters of the hand's 
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identifiedpalhs 250 and the hand's extracted motion com- conversion, step 658 finds a dioid activity slmctiire in a 
ponents from the motmn extraction module 16. If a slide of lookup table using a bitfield of the identities of either the 

" j"^".^ - '^^ fi"ge« °r fi"S«« in the synchronized subset 

orders slide mUration tests ^4 and 656. To distinguish slides Diflfeient finger identity combinations can refer to the same 
bom glancing finger taps dunng typing, decision diamond 5 chord activity structure. In the piefeired embodiment, all 
654 requires at least two fingers from a hand to be touching finger combinations with the same number of non-thumb 
the surface for sbde mode to start. TTkmc may be some fingertips refer to the same chord activity structure, so sUde 
exceptions to this rule such as aUowing a angle finger to chord activities arc distinguished by whether the thumb is 
resume a previous shde wthm a second or so after the touching and how many non-thumb fingers are toudiins 
previous shde diord hfts off the surface. lo Basing dwid action on the number of l^ertips rather th^ 

In a preferred embodiment, the user can start a slide and their combination stiU provides im to seven chords per hand 
specify its chord m either of two ways. Id the first way, the yet makes chords easier for the user to memorize and 
user starts with the hand floating above the surface, phces perform. Tht user has the freedom to choose and vary which 
some fingers on the siirface possibly asynchronously, and fingertips are used in chords requiring only one. two or three 
begms moving aU of these finge^ laterally. Decision dia- is fingertips. Given this freedom, users nadiraUy tend to pick 
mond 656 initial^ the slide mode only when significant combinations in which all touching fingertips are adjacent 
motion detected m all the touching fingers. Step 658 rather than combinations in which a finger such as the Kng 
selects the chord from the combination of fingers touching finger is lifted but the surrounding fingers such as the middle 
when significant moUon is detected, regaidtess of touch- and pinky must touch. One chord typing study found that 
down synctainization. In this case coherent initiation of 20 useis can tap these finger chords in which all pressed 
moUon m^ the touchmg fingers is sufficient to distinguish fingertips are adjacent twice as fast as other dioids. 
ttie sLde from restmg fingers, so synchronization of touch- TTie events in eadi chord activity stnicture are oiganized 
down is not necessary Also, novice users may erroneously into sKces. Each slice contains events to be generated in 
try to start a shde by placmg and sliding only one finger on response to motion in a particular range of speeds and 
toe sur&ce forgethng that multiple fingers are necessary. 25 directions within the extracted degrees of freedom For 
Tolerance of asynctoonous touchdown allows them to seam- example, a mouse cursor slice could be allocated any 
kssly conect this by aiteeqiienUy pUdng and sliding the translational speed and direction. However, text cursor 
rest of the fingeB deswd for the chord. TTie slide chord will manipulation requires tour slices, one for each arrow key. 
then miuate without forcmg the user to pick up all fingers and each arrow's slice integrates motion in a narrow direc- 
and aart over with synchromzed finger touchdowns. 30 tion range of translation. Each slice can also include motion 

In the second way, toe user starts wito multiple fingers sensitivity and so-called cursor acceleration parameters for 
rest^ on the surfiice, hfts asubset of these fingers, touches each degree of freedom. Tliese wiU be used to discretize 
l^"^ir .^"^ °° synchronoudy to select motion into the units such as arrow key clicks or mouse 

toe diord, and begins moving the subset laterally to initiate clicks expected by existing host computer systems 
toe shde^ Decmon diamond 656 actaaUy initiates the slide 35 Step 675 of chonl motion converrfon simply picks toe 
mode when it detects sigmficant motion in all the fingers of first slice in toe given chord activity stnKrture for processing 
the synchroiiized subset^ Whetoer the fingers which Step 676 scales toe current vahies of toe extracted vetocity 
remained restmg on toe surface during tois sequence begin components by toe shoe's motion sensitivity and accelera- 
k ^tell^Tn'r^ «« .k 1^ "^"^ parameters. Step 677 geometricaUy projects or clips toe 

^ detenmned m step 658 by toe combmation of fingers m 40 scaled velocity components into toe slice's defined weed 
tnLhi'n'"^ Tif"''" set of all and direction range. For toe example mouse cursorsHcS 

touclnng fingere. TTus seooiid way has toe advantage toat toe might only invoWe clipping toe rotation and scaling «m- 
user do« not have to hft toe whole band from toe surface ponents to zero. But for an arrow key sUce, toe traillation 
weth, H- ^^"^ """^ °f velocity vector is projected onto the unit vector pointing in 

weight of toe hands resting on toe surface and only lift and 45 toe same direction as toe arrow. Step 678 inte^tes ^ch 
press the Uvo or threefingeis necessary to identify toe most scaled and projected component velocity over time in the 
common finger chords. sKce's accumulators until decision diamond 680 determines 

To provide greater tolerance for accidental shifts in rest- at least one unit of motion has been accumulated. Step 682 
ingfingerpositions.deasiondianiond656requiresbotothat looks up toe slice's preferred mouse, key or three- 
aU relevant fingere are moving at signilcant speed and toat so dimensional input event format, attaches toe number of 
K aaumulated motion units to toe event, and step 684 di^ 

by threstolding tte geomet™ pauAes toe event to the outgoing queue of toe host com- 

^k1^ "^"^ ^ -^f ^ ""^^^ munication interface 20. Step 686 subtracts toe sent motion 

toe slow«t finger s sp«d is at least a mimmuxD fraction of events from toe accumulators, and step 688 optionally cleats 
toe fastest finger s speed. Once a chord slide is initiated, step ss toe accumulators of other sUccs. If toe slice is intended to 

fk. "^1 " g'^""""^ ^^e}'^ '«=y «>mm;md per hand motion, decision 

^ until eitoer the touchmg fingers or toe synced subset diamond 689 wfll detennine toat it is a one-shot shce so toat 



■ ... .... ^ , . step 690 can disable fiirtoer event generation from it until a 

Once the shde mitutes, the chord motion recoginizer shce wito a different direction intervenes. If the given sUce 

could sunply begm sendn^ raw component vetocities paired 60 is toe last dice, decision diamond 692 returns to step 650 to 

wito toe selected combination of finger identities to the host await the next scan of toe sensor array. Otherwise step 694 

HowevcMn toe mter^t of backward compatibility wito toe continues to integrate and convert the current motion for 

mouse and key event formats of conventional input devices, other shces. 

toe motion event ^neration steps in FIG. 40B convert Returning to HG. 40A. for some applications it may be 

inotion m any of the extracted degrees of freedom into ds desirable to change toe selected chord whenever an «Ui- 

standard mouse and key command events which depend on tional finger touches down or one of toe fingers in toe chord 

toe Identity of toe selected chord. To support such motion hfts off. However, in the preferred embodiment, Uie selected 
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chord cannot be changed after sUde initiation by asynchio- does not decay during motion continuation mode the host 

lr^fS^^T^J^t^''''^''^'T"^ computercanLd alignal instJ^cuTmodon «niS 
or hit additional fingers as may be necessaiy to get the best mode to be canceled if the cursor riches the edoe nf T 

S,'^°rii''"'^'t^r/^'=^°"-^~"'^P'^ screenorendofadoclfnt.Sarrif^;finX„^^^^ 
IS^^J^Z^^J'^.'^I^''''V^'^'^''''^ ' oothesurfaceduringmo.ion«.ntinuationXlS^^^ 

^^yt^L'Zt'^'^l ""^"^ ^ ^° alj^ U.e system to aUocatetasll^tweenLdsin^'m?^^ 

E L dlrSe^ ^ JT it ;^\f '° ^ triple fingertip chord on both hands. Tins way the 

f^l^T^?! u ? P^'^^^ly only 15 mouse pointer can be moved and drug with either hand on 

:&":r,^stS^j,tin^a^L^^r^ - ^^^-^-~^^^o.t^t-Z':^ 

St :^;fo ^tlo.TT.vl^'^^VK^ ^i*""* -face, while corresponding 

in^^H^^ 1 " »"°«'«» to Chords on the left hand For instLice left 

len^,^ i /iT^ con^nents resumes but the events fingertip pair movement would geL«te <lw ST-^om 
ST'be^dSett""' """'^ ^ --Js^-ponding to the direction of modon, Z ^e 

It is advantageous to provide visual or auditory feedback SS^ro^S" ^'^""^"'^ ^ ""^"^^"^ 

f"^"^ ^'^ ^ «»"P"te' systenis supporting manipulations in 

d^ ^^d^ r^hS if.h'T^' 'r^'^^' ^ detected. For example, the common edidn^«,m: 

Uo^ would be replaced with tone or tone burst combuui- mands cut, copy and paste can be inmitively allocated to a 

TTie accnmiilArinn ,nH , P""* scaling, chord tap. and anti-pinch hand scaling of 

Uon flag of a>e dominant motion component. The state of so layout which has ^en pi^-speciTed bftfe uLTS m^u 

e^Sn'^G m ^trl^L'^'r f 'T ^ layout «Sf a Z Jf l^^^on 
exm^ion CHG. 3^ if tansla&on is dominant, or by corre- structures. Each region has associated with it the swnbol or 
^ndmg flags "> step 534 of polar extraction. If there has commands which Luld be^to ±eC «m.u^ wh^n 
'^'!'"T- ^ "^""^ ^eion is pressed and coordhia^rrep^^Zle^ 

chord shde mode, setting the selected chord to nulL If the ss don of the Center of the region on In^e 
Dag mdicates no s.gn^.^t finger deceleration prior to prefen^l embodiment, arran^ment of t^^b^ rel^ 
l.Sr^'°° , 7^°"^ continuation containing alphanumeric and pun^iL s^teb 3w 

mode for the selected chord. Whfle in this mode, step 667 corresponds to either the OWFnTV or i^T ^ ^ 
applies the pre-lifloff wetted average (560) of dom'inant layouT^on 1 meS^^?^^^^,^""' '"'^ 
ZfTAI """""^ to the motion accumulators (678) in 60 In some embodiments of thTmiKS surface appara 
place of the current velocities, which are presumably zero dis it is advantageous to be able to smp or^^ T^v 

layouttothe.esl,gpositions1?A^.°^hr^^ 
mode does not stop until any of the remammg fingers not m helpful for multi-touch surfaces which are sever^timl^ 
^e synchrom^ subset are lifted or more fingen; newly larger than the standard keyboard oTS^la^u^a^^ 
touch down. This causes decision diamond 664 to become 65 covering an entire desk. Fbring the teTS iTonell^ 
fake and normal dide activity with the cunenUy selected fixed aL of such a suS^Jo^ L Snveint a^ 
chord to resume. THough the cursor or scrolling velocity discourage use of the whole ^^LbVe t 
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provide feedback to the user about changes in the position of right hand, and from the "A" key to the "F" key for the left 

the toy layout, the poaUon of the key symbols in these hand. Similariy, the thumbs can each be provided with a 

embodunents of the multi-touch surface would not be single oval-shaped depression at their default locations, 

printed pennanenUy on the surface. Instead, the position of slanted slighUy fiom vertical to match the default thui^ 
the key symbob would be tcprogrammably displayed on the s orientation. Hjese would preferably correspond to "Space" 

''•^H-'"!"^^ polymers, liquid cryOal, or other and "Backspace" key mgions for the right^ left thLbs, 

dymmic v«ial Asplay means embedded m the multi-touch respectively. Such minirial depressions^ uctflely guiS 

suAce apparatus along with the proximity sensor arrays. users' hands back to home row of the key layout without 

.h"'^ ^ "T"*!!^' "^'^^ '^'^ users to look down at the surface and without 

paths from both hands arid a_waits what wiU be known as a lO seriously disrupting finger chord sUdes and manipulaUons 

layout homing gesture. If decision diamond 704 decides on the surfkce 

with the help of a hand's synchronization detector that all The positions of key n:gions off home row can be marked 

lZ. J-^ h ^ i^f" "yP^ °f ^'^^'^ indicators. Simply roughening the 

i^'T^"". ^ '° "^"^ '«gi°-^ °°« ^* well. nio^Si humans 

layout to the hand such that the hand's home row keys lie is easfly differentiate textures when sliding lingers wer them, 

under the sy«:hronized fii^ertip^ wherever the hand is on most textures cannot be noticed during ^ick Ups on a 

^ ^"^K ^ ^- '^^""-^ "^"^ """^ '^'^'^ Only relatively abrupt edg« or p iions 

^^n^u,h h ^ f T"""!. u H^.'^ ^' can be sensed by the users' fingprti^ under typing condi- 

regions which are normaUy typed by the given hand in Uons. Therefore, a small raised doflike a Braillf dot is 

proporuon to the mei^jired hand offsets. Note the currenUy 20 formed on lop of the surface at the center of each key region 

measuredratherthanfflterrfestmiat^ lie user receives feedback on the.accuracy of tSkS 

because when all five fingprs are down there is no danger of strokes fiom where on the fingertip a dot is felt Th^ 

T^^'J!!^^ corrupting the measured ofi&ets. feedback can be used to correct finger aim during fiituie 

Thisprocedure assumes that the untranslated locations of the keypresses. Since single finger slid^ are ignored by the 

home row toys are the same as the default fiiger locations 25 chord motion recognizer, the user can also sHde a finger 

lor me liana ^ ^^^^^ ^ particular key 

Decu(K,n diainond 708 checks whether the fingers appear region's dot and then tap the key mgion when the dot I 

t , \°T^^Tf^ ^ ^"'^'^ '«>1^ 'he surfa«. Each dot should be 

outaretched or pmch^ together. If the posture is close to just large enough to be felt during topping but not so large 
neutral^ ^p 710 may farther offset the keys nonnafiy typed 30 as to impede chord slides across the sif a J. Even tf^^ 

by each finger, which for the most part a« the keys in the are not large enough to impede sliding, they can still corrupt 

^„n?^r f ^^"fi''^ the measured finger ofl&els. proximity and fingertip c^id mJLrxiients by raisnfg 

Temporal filtermg of these finger oJbels overseveral layout the fingertip flesh near the dot off the surface thiB locally 

hommg geshires will tend to scale the spacing between separating the flesh from the underiying proximity sensing 
columns of keys to the user's hand size. Spacing between 35 electrode. Tlierefoie, in the preferredVmtodiment^eT^ 

rows IS scaled down m proportion to the scalii^ between tica of each dot above the surface diele<aric is made of a 

"'u^i,. 1, 1 . r .u u J, . . . conductive material. This improves capadtive coupling 

With the key layout for the band's keys moiphed to fit the between the raised fingertip fl^b and the underiying elec- 

size and current position of the resting hand, step 712 trades. 

^ steps within the keypress detection 

^etLl'„"i;'1^ l^*" '^f.^" ^'^T *^ '°«P- '^'rieves torn the current idemified path d^a 

^ ^ ^ •^J^^?"* user can begin 250 any paths which were recently created due to hand part 

to type and the typing recopuzer 718 wdl use the morphed touchdown or the surface. Decision diamond 752 chwks 

key region tocations to decide what key r^ons are being whether the path proximity reached a keypress proximity 

presed The layout will remam morphed this way until 45 thresh for die first time during the current «L,r aLy scan 

either toe user performs another homing gesture to move it If the proximity has not readied the threshold yet or has 

T^T^u J" ' ^ ^"'^ ^^"^'^ P«=viousIy. contral returns to s^ 750 to 

^^t, n ? .u '^B^'^ d^«l 714 will try keypressdetection on toe next recent pato. If toe path just 

evenhuUy tmie out so that step 716 can reset toe kyout to crossed toe keypress praximity toreshokf decision dlamo^ 

lis default position in readmess for another user or usage 50 754 checks whether toe contact pato has been identified as 

„ ,. , , afiugerralhertoan a pahn. To give the users the freedom rest 
. For smaller mulu-touch surfaces in which toe key layout toe pahns anywhere In toe surface, pali pr^^esstoSd no! 

is peimanenUy printed on the surface, it is advantageous to nonnally cause keypresses, and are toerefote ignored 

give toe user tactUe feedback about toe positions of key Assuming toe pato is a fi^ decision diarJ^nS 75^ 
regions. However any toctde mdicators placed on toe sur- 55 whetoer toe hand toe identified finger comes from is cur- 

f^ must be «refaUy designed so as not to impede smooto renlly performing a diord slide gesti^e or writing via toe pen 

sUding across toe surface. For example, shallow depressions grip hand configuration. Asyn^hranous finger presses ^ 

made m toe surface near toe center of each key mimicking ignored once these activities have started, as also indicated 

toe shallow dqjiessions common on mechanical keyboard in step 660 of FIG. 40A Assuming such hand activities are 

l7nT!^/° '^T •'ffe*^ as toe 60 not ongoing, decision diamond 757 proceeds wito debounce 

h^dshdes across toe surface To mmimize sudi washboard tests which dieck toat toe finger has toudied the surface for 

nSef f^Mif "fi w '""^.f w '^^ a' least two sensor array scan cycles and toat it had been off 

provides for toe fin^ips of each hand a single, continuous toe surface for several scan cycles before touching down 
depresaon tunning from toe default index fingertip tocation The pato traddng module (HG. 22) faciUtotes aid. liftoff 
L n^^T^i ^"^^ ^n*^"- ?f ~'^'«1* "n « debouncing by reactivating in step 334 a finger's old pato if 
toe QWEimr key layout to shallow. shghUy arched chan- toe finger Ufls off and quickly toud«s badfdown o^r toe 
nels along home raw from toe "r key to toe key for toe same spot. Upon reactivaUon toe time stamp of toe last 
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^rr^h^'°"'^'*^'*^^'««>« ''""be newest ' "^^^ f"*''* "sponsc to keypresses is weU kno^ 

r^K*iUunareasonabledistanceofthefinger.andifno. touchscreeos, kiosks, appUauce »ZS 

^^^^%^'°*^:^°'^^'''^^^^"'&on P^e'^.'U'd mechanical keyboards in whiS die keSS. 

!2 ""^ '^^^ a keypress clement "^^^y sOent and docs not have a maK!^ 

fdcnul^,^^^ ^''^ ISge which feels distinctive to^c i^r pj^k^ 

S 7« t&!^^^h- P'^'^y '^^W^ e^ » keypress iT senrSr 

o^tTCr '° J~'rSrS;h='g L°Sf " sy^hol 

TTie keypress queue efifectivelv orden^ . J^l.'*'* ^'^ ""^ y*"' •'^e^' decision diamond 786 

by when t^y pa^ U«= kf^ jSy'SSt.j:^ L^i^S. '^''^J^^y -S^- - ^ modifier 1^ 

fixes the order in which key symbols fiomM^Sairt^ ^xTJ et^nT^^K^' If so, step 788 advances to the 

will be transmitted to the host. However, an ekmStev ^i„! "n? "h^*" '^^'^ ^'^^ '^"^^'^'^ *e head. Pro- 
symbol ,s not assured transmission of the hoj?^^'^^ ^ ^^jJir ?;.K'''P^'°*''^"«^ element 

keypress queue. Any of a number of conditions sX l!^^^- '»'«=essfully reaches 

bemg part of a synchronized subset of pressing c^ fetn^^'"" ^ "^"^ ^ack toVard the 

^Th'I queue bcforelS^ ^^^r ^7'^^'' '"Sions which are still 

mitted to the host In this sense the keypress que^sS S^ ,^ ^ J** can send the next element's key 
be considered a keypress candidate queue. Sc ie 25 '^'^^ ^""^ '^'^ """^^ ^bols of any preceding 
ordered lists of finger touchdowns and release mSiiS^ ""^""^ 

foreachhandseparalelyintbesynd«,niza^dSrS o^'^f'""/^""^ ^« -1>^'^ th=- «-ers touch th^ 

keypres. queue includes and orders the finger LSL^T ^ ^u^'^ °^ few huXl 

from both hands. ^ 'oucnoowns ™Uiseconds for a key to be sent Hiis liftoff limine reanire 

HG.43Ashows the steps within the keypress acceptance 30 m^L^.^f'v'" "•^elhold of 

and transmission toop. Step 770 picks the elem^K^ ^ u force threshold of 

head of the keypress queue, whid. represenr.te o d^! wa^fS ^"^T'^*''': '^^ constraint provides a 

finger touchdown which has neither b^n deleted &om^ ^tohW , t ' '° "^i^ ^''8'='°" without 
queue as an invalid keypress candidate nor transrS^ ft^ Zl^.l^"^- ^ ^y^ehronization detector 14 pro- 
assomted key symbol. Decision diamond 35 ^JT ^' ^.^^eis to rest on the surface wittout 

whether the path is stfll identified as a finger wS ^.^^ '^"y """^ down at the same 

jn «,e queue path pioximity could hav^crea^d J mS S S ^ 
Jat the Identification system decides the path is^TlJv fi 5 ''^ amultaneously placing the central 

&om a palm heel, in which case step T«^d^le^ J-gertips o„ t^ s^ce, bu, then they foUow asyncta^ 

teypie^ element without tnmsmittingt^, the host and sS « ^"^"^^"^ '^'^"'^'^^ ''^"^ 

7 't'^ processing to the next element Deci^ ^ ^n-t^ T^^'^'^'^'^'^y'^y^''^'"'^ 
dumond 774 also invalidates the element if its pre^^ ^ Tt J "T^^^ed by the synduonization detector, 
pened synchronously with other fingers of the Sl^i^K^ "l"' """Ple hundred 

TTius decBKjn diamond 774 foUows through on de^on "^"'"^decBion diamond 782 wiU delete them with- 
command steps 601^ 612. 615. 620 of the s?Sc^^^S 45 tot^o^™ H ^ P^-^es 
detection process (HG. 39). Decision diamond 776™? 

dates the keypress if too much lateral finger motion^ S^n^^?^'' * "^'^ ""P' cause 

occurred sux* touchdown, even if that latoS finger motio^ £c^ ^.1 " 1?'^ •<» «f 

has not yet caused a chord slide to start. Because usermay ^nZ °° "^"^ '° hands while 

be touch typmg on the surface, several millimeters^S 50 ^L^T °^ 'y^^^' '^^ "^ers to 

mouon are aUowed to accommodate glandne fiLS ^ ;f °^ diem above the 

moUonswhichoftenoccurwhenquiddySSf^^ff ^ ^ 'yP-g fences. This is acceptaW^ 

This^ much more glancing tap motion tfian iT^eS^rv poshire except that the users arms will eventuaUy tir^ 

^hpads which employ a single finger ^IT^^ ^itn^^'^'^-'^e hands bade on the surface between 
cursor mampulation and a single firmer tap for kev or mon^ « ^ 

button elide emulation. '«Pl°''«y°r mouse ss To provide an alternative typing posture which does not 

Decision diamond 780 checks whether the finger whose l^ZfL^^"" ^. "«= deci- 

touchdowna«,,edU,ekeypressekmenthassince\-f^^ whfchd^nn. ""de 
the surface. If so. decision diamond 782 checks whether Ia,te,a T ^^er liftoff after eadi press. 

washft^^ffsoonenoughtoqualifyasanormalS^S iJ 60 So ^^I'^'^^'^f!" '^'^ fi°ge,^ of a hand 

so. step 784 transmits the associated key symbol toti^tZ eadi time a finger is asynchro- 

and step 778 deletes it from the head oTtt.^uSl'^t S Sy'S '^^.Tj^'^ P^-dTn a key'c^t . 
a keypress us always deleted from the queue upon liftoffin ml^ l "^^"^ .T^u** '^'"ed regardless of subseqwn 
even though it may have stayed on Ic su^k^,^ ^LT^- m '^'^ ^ "-at ^ertj 

exceeding the tap timeout, it may have stiU «uS tiJ^ « ^^^-rS^^ ^'^'^ is applied, dedsZ 

~asamodifierkey,asan^pulsiv^^"S.S " t^Z'^'tt'''^'^'°"''^^'y^"'^'^ 
resting, or as a typemalic press, as described betow 'rofir^^^r '^ '^*?"8erproxmuty peaks. If the proximity 

profile mcreases to Its peak very slowly over time no key 
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finger with the mtenUon of generating a key but then Step 812 recoJ^ the time U.e key^K Sor S 

SSLrd'T^ touch surface is a>n3pcessible, reference by decision diamond 8^'^'^^^^^ 

dec^ c^ diamond more direcaymfer^^^^ 5 to step 770 to await the next proximity image scan 

&om the ratio of measured fingertip proximity to ellipse axis UntU the finger lifts off or another iaps alynchronously 

lengths. T^en it can threshold the inferred force to distin- procesang will pass through dedaon diamond 798 to chedk 

^ deliberate key presses from gentle finger rests. Since whether the key symbol should be sent again Step 806 

when mtendmg to generate a key the user will normally computes the symbol repeat interval dynamically to be 

pre^ down on the new key region quickly after lifting off the lo "aversely proportional to finger proximity. Thus the key wiU 

old key region the mipulsivity and force thresholds should repeat faster as the finger is pressed on the surface harder or 

mCTease with the time ance the finger lifted off the surface. a larger part of the fingertip toudies the surface This also 

tmulatmg typematic on a multi-toudi surface presents rcAices the chance that the user wiU cause more repeats than 

speaal problems if finger resting force cannot be distin- intended since as finger proximity begins to drop durine 

guished rehably from sustained holding force on a key is the repeat interval becomes much longer Decisis 

region. In th^ case, the ^cial touch timing sequence diamond 808 checks whether the dynamic repeat interval 

detected by the steps of FIG. 43B supports reUable type- since the last typematic symbol send has cla^d and if 

mauc emulation. Assummg decision diamond 798 finds that necessary sends the symbol again in 810 and updates the 

typematic hasn't started yet, decision diamond 794 checks typematic send time stamp 812. 

whether the keypress queue element being processed repre- 20 't is dearable to let the users rest the other fingers back 

sents the most recent finger touchdown on the surface. If any onto the surface after typematic has initiated 804 and whfle 

^^Ii 1°''';^°^ ^^""^ foUowed the toudidown repre- typematic continues, but the user must do so without Up- 

sented by this element, typematic can never start firom this P^^ Decision diamond 805 causes typematic to be canceled 

queue element. Instead, decision diamond 796 checks ^nd the typematic element deleted 778 if the user asynchro- 
whether the element's fiiiger has been touching longer than 25 ^^^^Y taps another finger on the surface as if trying to hit 

the normal tap time out If the finger has been touching too «»other key. If this does not occur, decision diamond 782 

long, step 778 should delete its keypress element because ^1 eventually cause deletion of the typematic element 

decision diamond 786 has detennined it is not a modifier and when its finger lifts off. 

dedsion diamond 794 has determined it can never start The typing recognition process described above thus 
typematic. If decision diamond 794 determines that the 30 a^ows the multi-touch surface to ergonomically emulate 

keypress element does not represent the most recent both the typing and hand resting capabilities of a standard 

touchdown, yet decision diamond 796 indicates the element mechanical keyboard. Crisp taps or impulsive presses on the 

has not exceeded the Up tune out, procesar^ returns to step saifacc generate key symbols as soon as the fineer is 

77U to await either hftoff or timeout in a future sensor array released or decision diamond 792 verifies the impul^ has 
scan llus aUows finger taps to overlap in the sense that a 35 peaked, ensuring prompt feedback to the user Pincers 

new key re^on can be pressed by a finger before another intended to rest on the surface generate no keys as lo^ as 

tmger lifts off the previous key region. However, either the they are members of a synchronized finger press or release 

press times or release Umes of such a pair of overlapping subset or are placed on the surface gendy and remain there 

nnger taps must be asynchronous to prevent the pair from aloi^ with other fin^rs for a second or two Once resline 
bemg considered a chord tap. ^ fingers can be lifted and tapped or impulsively pressed on 

AsKiming the finger touchdown is the most recent, ded- the surface to generate key symbols without having to lift 

sion diamond 800 checks whether the finger has been other resting fingers. Typematic is initiated ether by impul- 

touching for a typematic hold setup interval of between s^^ely pressing and maintaining distinguishable force on a 

about half a second and a second If not, processing returns or by holding a finger on a key while other fingers on 
to 770 to await either finger liftoff or the hold setup 45 the hand are lifted. Glancing motions of single finlers as 

^ T ""^^ °^ ^ "^^^"^ ^ ^i^y tolerated since most cursor 

wnen me hold setup condition is met, decision diamond 802 manipulation must be initiated by synchronized slides of two 

checks whether all other fingers on the hand of the given o*" more fingers. 

finger keypre^ lifted off the surface more than a hatf second Other embodiments of the invention will be apparent to 
ago. It they did, step 804 wiU initiahze typematic for the so those dolled in the art from consideration of the spedfica- 

^^^^ oJS^ combination of dedaon dia- tion and practice of the invention disclosed herein It is 

inonds 800 and 802 aUow the user to have other fing^ of intended that the specification and examples be considered 

me hand to be resting on the surface when a finger intended ^s exemplary only, with a true scope and spirit of the 

tor typematic touches down. But typematic will not start invention being indicated by the following claims 
unless the other fingers lift off the surface within hatf a 55 What is claimed is: 

second of the desired typematic finger's touchdown, and 1- A sensing device that is sensitive to changes in seff- 

typematic will also not start until the typematic finger has a capadtance brought about by changes in proximitv of a 

<x,nlmued to touch the ^ toudi device to the sensing device, the sensing device 

toe omers lifted off the surface. If these stringent conditions comprising: 

are not met, the keypress element will not start typematic eo two electrical switohing means connected together in 

^.TlT'^lt''^^^^ '"^^ ^ ^^^^ an^ncSf^' ^ 

when me finger hfts off or through Up timeout 796) if output node; 
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an integrating capacitor to accumulate chaige transfened 
during multiple consecutive switdiing of the series 
connected switching means; 

another switching means connected in parallel across the 
integrating capacitor to deplete its residual charge; and 

a vollage-to-volUgp trandation device connected to the 
output node of the series-connected switching means 
which produces a voltage representative of the prox- 
imity of the touch device to the sensing device. 

2. The senang device of claim 1, wherein the electrical 
switching means comprise semiconductor transistors that 
are switched on and off by cfedicaled control circuitry. 

3. By The sensing device of claim 1, wherein the electrical 
switching means comprise polymer transistors that are 
switched on and off by dedicated control circuitry. 

4. The senang device of claim 1, wherein the electrical 
switching means comprise thin film transistors that are 
switched on and off by dedicated control circuitry. 

5. A sensing device thai is sensitive to changes in seff- 
capacitaiKre brought about by changes in proximity of a 
touch device to the sensing device, the sensing device 
comprising: 

two electrical switching means connected together in 
series having a common node, an input node, and an 
output node; 

a dielectric-covered sensing electrode connected to the 
conunon node between the two switching means; 

a power supply providing an approximately constant 
voltage connected to the input node of the series- jq 
cormected switching means; and 

an integrating current-to-voltage translation device con- 
nected to the output node of the series connected 
switching means, the currcnt-to-voltage translation 
device producing a voltage representative of the prox- 35 
imity of the touch device to the senang device. 

6. A sensing device that is sensitive to dianges in self- 
c^acitance brought about by changes in proximity of a 
touch device to the sensing device, the sensing device 
comprising: ^ 

two electrical switching means connected together in 
series having a common node an input node, and an 
output node, 

a dielectric-covered senang electrode connected to the 
connmon node between the two switching means; and 45 

a power supply providing an ^proximately constant 
volUge connected to the input node of the series- 
connected switching means. 

7. A multi-touch surface apparatus for detecting a spatial 
arrangement of multiple touch devices on or near the surface 
of the multi-touch aj^aratus comprising: 

one of a rigid or flexible surface; 
a two-dimensional array of the sensing devices of claim 1 
arranged on the surface with their output nodes con- 
nected together and sharing the same integrating 
capacitor, charge depletion switch, and voltage-to- 
voltage translation device; 
control circuitry for sequentially enabling each of the 

sensor devices; 
voltage measurement circuitry to convert sensor data to a 

digital code; and 
circuitry for communicating the digital code to another 
electronic device 
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9. The multi-toudi surface apparatus of daim 7, wherein 
the surface comprises a micro-<limensional surface. 

10. The multi-toudi surface apparatus of claim 7 being 
one of fabricated on or integrated with a display device. 

U. The multi-touch surface apparatus of claim 10, 
wherein the display device comprises one of a liquid crystal 
display (LCD) or a light-emitting polymer display (LPD). 

12. A multi-layer cover apparatus for the multi-touch 
surface apparatus of claim 7, comprising: 

a compliant dielectric layer; 

a deformable conductive layer formed on the dielectric 
layer, the conductive layer being electrically coupled to 
the voltage or current measurement device; and 
a touch layer formed on the conductive layer. 

13. The multi-layer concr apparatus of claim 12, wherein 
the touch layer has a symbol set printed thereon that can be 
removed and replaced with an alternative symbol set 

14. The multi-touch surface apparatus of claim 7, wherein 
the apparatus is ergonomically arched. 

15. The multi-touch surface apparatus of claim 7, wherein 
the apparatus includes hand configuration visual indicators. 

16. The multi-touch surface apparatus of claim 7, wherein 
the apparatus inchides hand configuration audio indicators. 

17. A multi-touch surface apparatus for detecting a spatial 
arrangement of multiple touch devices on or near the surface 
of the multi-touch apparatus comprising: 

one of a rigid or flexible surface; 
a two-dimensional array of the sensing devices of claim 2 
arranged on the surface with their output nodes con- 
nected together and sharing the same current-to-voltage 
translation device; 

control circuitry for sequentially enabling each of the 
sensor devices; 

voltage measurement circuitry to convert sensor data to a 
digital code; and 

circuitry for communicating the digital cotte to another 
electronic device. 

18. A multi-touch surface apparatus for detecting a spatial 
arrangement of multiple touch devices on or near the surface 
of the multi-touch apparatus comprising: 

one of a rigid or flexible surface; 
a plurality of two-dimensional arrays of the sensing 
devices of claim 1 arranged on the surface in groups 
wherein the sensing devices within one group have 
their output nodes connected to corresponding sensing 
devices within other groiq)s and share the same inte- 
grating capacitor, chaige depletion switch, and voltage- 
to-voltage translation circuitry; 
control drcuitry for enabling a single sensor device &om 

each two-dimensional array; 
means for selecting the sensor voltage data &om each 
two-dimensional array; 

voltage measurement circuitry to convert sensor voltage 
data to a digital code; and 

circuitry for communicating the digital code to another 
electronic device. 

19. The multi-touch surface apparatus of claim 18, 
wherein the sensor voltage data selecting means comprises 
one of a multiplexing circuitry and a phirality of voltage 
measurement circuits. 

20. A multi-touch surface apparatus for detecting a spalial 



— — ^« „^ , ^v. ujuiu-iuucD sunace apparams ror oetecting a spatial 

8. A phirality of the multi-touch surface ^aratuses of 65 arrangementofmultqjle touch devices on or near the surface 
claim 7 arranged in the shape of one of a cube, a sphere, or of tbc multi-touch apparatus comprising: 
any other three dimensional sh^. one of a rigid or flexible airface; 
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^ oJ^^a^-^^ ^^"^"^ ''^^'^ * arranged on 

control circuitry for sequentiaUy enabling each of the 
sensor devices; 

voltage measurement circuitry to convert sensor data to a s 
digital code; and 

circuitry for communicating the digital code to another 

electromc device. 
21. A rnulti-touch surface apparatus for sensing diverse 

configurationsandactivitiesoftou<*devicesandgeneratinp ^ 
integrated manual input to one of an electronic or electro- 
mediamcal device, the apparatus comprising: 

an array of the proximity sensing devices of claim 1* 
a dielectric cover having symbols printed thereon that 
represent action-to-be-taken when engaged by the is 
touch devices; 

sc^g means for forming digital proximity images 
from the array of sensing devices; 

cahbrating means for removing background offsets from 
the proxiimty images; 

recognition means for interpreting the configurations and 
activities of the touch devices that make up the prox- 
imity images; 

processing means for generating input signals in response 
to particular touch device configurations and motions; 
and 

communication means for sending the input signals to the 
electronic or electro-mechanical device, 

22. The multi-touch surface apparatus of claim 21 
wherem the symbols printed on the dielectric cover can be' 
removed and replaced with an alternative symbol set 

23. The multi-touch surface apparatus of claim 21 
wherem the dielectric cover has conductive fibers therein' 
the conductive fibers being oriented normal to the array of 
sensing devices for conducting the capadtive ettect of the 
touch devices on the array of sensing devices. 

24. A multi-layer cover apparatus for the multi-touch 
surface apparatus of claim 21, the multi-layer cover appa- 
ratus comprising: 

a compliant dielectric layei; 

a deformable conductive layer formed on the dielectric 
layer, the conductive layer being electrically coupled to 
the voltage or current measurement device; and 

a touch layer formed on the conductive layer' 

25. The multi-touch surface apparatus of claim 21 
wherem the apparatus is ergonomically arched. 

26. The multi-touch surface apparatus of claim 21 
wherem the apparatus inchides hand configuration visual' 
mdicators. 

27. The multi-touch surface apparatus of claim 21 
Wherem the apparatus includes hand configuration audio' 
mdicators. 

28. The multi-touch surface apparatus of claim 21 being 
°°?D° -^"^^ °° integrated with a display device 

29. The multi-touch surface apparatus of claim 28 
wherem the display device comprises one of a hquid crystal 
dismay (LCD) or a light-emitting polymer display (LPD) 

30. A multi-touch surface apparatus for sensing diverse 
configurations and activities of fingers and palms of one or 
more hands near the surface and generating integrated 
manual mput to one of an electronic or electro-mechanical 
device, the apparatus comprising: 

an array of proximity sensing means embedded in the 
surface; 

sc^g means for forming digital proximity images 
&om the proximities measured by the sensing means; 
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im^e segmenution means for coUecting into groups 
those proximity image pixels intensified by contact of 
the same distinguishable part of a hand; 
contact tracking means for parameterizing hand contact 
features and trajectories as the conUcts move across 
successive proximity images; 
contact identification means for determining which hand 
and u^ich part of the hand is causing each surface 
contact; 

synchronization detection means for identifying subsets 
of identified contacts which touchdown or liftoff the 
surfece at approximately the same time, and for gen- 
eratmg command signals in response to synchronous 
taps of multiple fingers on the surface; 
typing recognition means for generating intended key 

symbols from asynchronous finger taps; 
motion component extraction means for compressing 
multiple degrees of freedom of multiple fingers into 
^gnees of freedom common in two and three dimen- 
aonal grq)hical manipulation; 
chord motion recognition means for generating one of 
command and cursor manipulation signals in response 
to motion in one or more extracted degrees of freedom 
by a selected combination of fingers; 
pen grip detection means for recognizing contact arrange- 
ments which resemble the configuration of the hand 
when grippmg a pen, generating inking signals from 
motions of the inner fingers, and generating cursor 
inampulation signals from motions of the palms whHe 
the mner fingers are lifted; and 
cornmunication means for sending the sensed configura- 
tions and activities of finger and palms to one of the 
electronic or electro-mechanical device. 
31. A multi-touch surface apparatus for sensing diverse 
configurations and activities of fingers and palms of one or 
more hands near the surface and generating integrated 
manual input to one of an electronic or electrx)-mechMiicai 
device, the apparatus comprising: 

an array of proximity sensing means embedded in the 
surface; 

sc^g means for forming digital proximity images 

trom proxmuties measured by the sensing means; 
contact segmentation means for collecting proximity 
image pixels caused by the same hand part into groups- 
contact tracking means for parameterizing hand contact 
features and trajectories as the contacts move across 
successive proximity images; 
contact identification means for determining which hand 
and which part of the hand is causing each surface 
contact; - 

synchronization detection means for identifying subsets 
of identified conUcts which touchdown or liftoff the 
airface at approximately the same time; 
typing recognition means for generating' intended key 

symbols from asynchronous finger Ups; 
motion component extraction means for compressing die 
dozens of degrees of freedom in motions of multiple 
fingers into the degrees of freedom common in two ^ 
three dimensional graphical man4)ulation; 
chord motion recognition means for generating, command 
or cursor mampulation signals in response to motion in 
one or more extracted degrees of freedom by a selected 
combination of fingers; and 
communication means for sending said generated input 
signals to the electronic or electro-mechanical device. 
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more hands near the surface and geneming integrated ^"^T ^^^^ filling an 

manual input to one of an electronic or electn^mechlm-^ ellipse to each group of pixels; and 

device, the apparatus comprising: updating positions of the segmentation regions of the 

an array of proximity sensing means embedded in the ^ f"'^'*""^PO"setofiirther analysis of the position and 

surface; ^ matures extracted from eadi group of pixels 

scMning means for forming digital proximitv imaccs ^^^"J''^f?i'^?'^"^ sloi^ysegmenution 

from proximities measured by U« s^^ZZ^ ?.^Z,°^ ^ F""^ ^^&>i^ areas under 

contact segmentation means for coUecti^ pio^tv ,n .if . T^?'^ ""'^ P^Md^'y valleys 

miage pixels caused by the same hand partlnfo^^^ '° ' ^'J^^ " background signd 

contact tracking means for parameterizing hand contac ■ ."T"*^"^ ^"'^ a remaining image 

features and trajectories is the coS mTe^!^ fT7 ' «'"«.«em'=«rtaUon region that estabUshes groi, 

successive proximity imagps; ooundanes at a directional proximity minima encounlered in 

contact identification means for determining which hand °^ «=arch outward from a given local maximum 

and which part of the hand is causing each surface IS hv n,..i^ f , • ^ 

contact; ^ ^ .■**T'*°*"*^°f«='»''°^'^remsegmenUtion region 

pen grip detection means for recognizing conUct arrance f '^4°^ ""^"^ segmentation 

ments ^idi resemble the co^Lo T^c^^ f ml'od^ d'llS'^^^ 

when gnppmg a pen, generating inking signals from 20 of each^n 1 ^ 34 wfaerem verified properties 

mouons of the inner fingers. Li generaSrcw^r S^td^^ H ^i"^""^ ^'^^ ^ as 

manipulation signals from motions of the^atos^e f "^'^ *° °f *g°«=°- 

theimier fingers are lifted; and "'^P'^'*^^ tatoon regions for the current image. ^ 

communication means for sending said eenerated inn.,t ^r^.J. associating into paths those surface 

typing, multiple degree^freedom man7pSortS "f ""^ fi""" 'heir 

chords, and handwdting. the method compriskS ^st^ 30 fi 7 °? '""^ 

P g me steps 30 fi^drng for each of a group of pixels in current proximity 

segmenting each proximity image into groups of elec- S„„ ""^^ ' p:^dictcd path 

trodes which indicate significant proximity, each group fi„S^„ r u ■ • 

representing proximity ofa distinguishable hand plrt or Tf ■J -^'^^ Path the pixel group whose 

other touch device; ^ 35 closest to the predicted path poation and 

extracting total proximity, position, shape, size, and ori- ^ ' path-dependent tracking 

entation parameters fiom each group of electrodes- • • 

tracking, group paths through successive proxiniitv ""^^ • F . ^ path if the pixel 

images inchding detection of path endp^Lts a^^^ P«^l ^roup to the path; 

touchdown and liftoff 40 *'^"mg new paths for remaining unpaired pixel groups; 

computing velocity and filtered position vectors alone ''''^S'h^'T P"'^ 7*"* ""ave no pairable pixel groups 

each path; ^ 'vithin the path-dependent tracking radius; and 

asagning a hand and finger identity to each contact oath "P^^jpS Pa'h parameters from the measured parameters 

by incorporating relative path positions and velocMes. a i ^^"^ "^"^ P^^- 

individual contact feanires, and previous estimates of offc^. fT computing hand and finger position 

hand and finger positions; and measured positions of individual hand 

maintaining estimates of hand and finger positions from "^t^XZ^ multi-touch sjrface for the purpose of biasing 

trajectories of paths currenUy assi^ to ^t^Z Jfl ~ identifications or morphing the k^ 

wherein the estimates provide hi^Tvel feedbaf to X^tog Oie S^- 

bias segmentations and identifications in future images *° «'"P™»g«'e steps o£ 
34. A method for filtering and segmenting hand contacts ««ablishing fingertip, thumb, or palm identities for each 
m a sequence of proximity images in order to support ...T v ' 

mterpreution of various conUct sizes, shapes, orientation. estabhshu^ an ofl&et weighting for each contact; 

and spacmgs, the method comprising the steps of: computing a hand position offset, wherein the ofeet is a 

creatir« a smoothed copy of the most recent proximity " weighted average of the diflFerence between a measured 
"nage; ' posiUon of each contact and a predetermined default 

searching for pixels with locally maximum proximitv in "I ^ corre^nds to an 

the smoothed proximity image; estabhshed uientity of the contact; and 

searching outward from each lo<^al maximum pixel for «, "'"P"^? a finger position ofiEset by subtracting a pre- 
contact boundary pixels using boundary tests of pixel *tenmned default position of an associated hand part 

and neighboring pixel proximities which depend on «>ntact and the hand position oflfeet from a 

properties of hand contacts expected in a segmentation 40'°'^ .1!!^"'?°.°^ the contact, 
region of the pixel; ^ "f claim 39. wherein conservative esti- 

forming groups from those pixels surrounding each local 65 f Jl^ ^ maintained 
maximum pixel up to Ld including thf b^undS ^Ji^l^T ^'^^ ^and 

pixels; uimig me Doundary contact identifications become unreliable, the method com- 

prising the steps of: 
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Jar hand is rotated and/or scaled to matdi a previous esti- 
computing a weighted average of the individual hand orientation and size of that particular hand 

contact velocities; 5 49. The method of claim 42, wherein aU of the surface 

predicting a current hand and finger ofiset from the ^ assigned are assumed to have come from the 

previous oflfeet estimates and the weighted averaee ^^°*^d and aU of the attractor points represent parts of 

velocity ®^ hand. 

computing current hand and finger offsets from o.rrPn. ih.^*/^?- "^^^"^ °^ v^CTcin an additional hand 

seuing wc currenUy measured hand and finger ofl&ets to means assigned to a particular hand 

zero if the hand has no contacts in the current image; ^1. The method of claim 42, wherein the measured 

distance from a contact to a thumb attractor point on a given 

updating the hand and finger oflfeet estimates to the 15 .f'^ ^ ^ ^ ^ encourage assignment to the thumb 

weighted average of the predicted offsets and currenUy ^f^^^. total contact proximity is greater than that 

measured offsets, wherein the relative weiahtine eiven ° fingertip but less than that of a typical palm heel, 

to the measured oflfeets increases in proportion to the ^iZIn^L ""^"^ "^'"^^ ^ measured 

confidence level in the current identificado^ * "^"'^^ ^ ^ ^"^^ heel 

4LTTiemethodofcIaim4nhlSaSSsH^^ ,n ""^^-^ ^^"'^ ^and is weighted so as to encour- 

^2 A ^PthLi ? f^:- u- . - . ^ given hand is weighted so as to encourage assigniient to the 

42. A method for estabbshmg identities of hand conUcts Palm heel attractor when the measured <»ntact width or rado 

on a mulU-touch surface usmg relative contact positions and ^tal proximity to eccentricity exceeds that of a tvoical 

features, the method comprising the steps of: finger. 

defining a template of hand part attractor points on the ^^^od of claim 42, wherein additional contact 

surface, the attractor points for each hand rouehlv 30 "^^^^ntact features are incorporated during a verifi- 

forming a ring ^^0° step, the venfication step comprising shifting assign- 

iiiacior point, ture tests used m the verification step. 

weighung the distances between each surface contact and 55, The method of claim 54, wherein the verification step 

each attractor point according to bow closely measured horizontal position coordinates of contacts assigned 

contact features sudi as proximity to a surface, shape, ^ attractor points corresponding to fingertips to ensure they 

aze, eccentricity, orientation, distance to nearest neigh- ^ ^ increasing order for right hand fingertips and in 

bor contact, and velocity match features typical of the decreasing order for left hand fingertqs. 

hand part the attractor point represents; . ^J^- method of claim 54, wherein the verificaU'on step 

finding a one-to-one mapping of the surface contacts to ^ ^ * ^"^^ verification step comprising the following 

the attractor points that minimizes a sum of distances fin^;^ , • 

between each surfece contact and its corresponding S^JZThT ^^il^"'"^ "^^^^ ^ 

attractor point; and ^ contact assigned to a fiUed attractor point correspond- 

ing to an innermost finger; 

re^gmzmg particular hand configurations from the num- 45 computing a thumb faaor as a fimction of the innermnct 

43 Se m^ ofct^^.Tl'""^- ^'^^^ ^^g- ~ ^ aTSlttractor 

44. nie method of claim 42. wherein the distance mefnV ^T^"^ ™"™?*' tP a fingertip attrac- 
used forcomputiog the dis.a.o;^rrs^a^i^ T,^ . T .TT"^ "^""^ ^ 
attractor point is the squared EudideanlS^ ^-gned to the thumb attnictor poi^^ and the thumb 

45. Tie method of daim 42, whe^ tS^a^actor point 55 57 £ 1^^');^! ^'^^'T'^- 

positions correspond to the positions of hand p^^a« hi^ if the oJe^t^l^"!^^^- ^ 
contacts measured when each hahH , i ~, " " "nenlalion and size of the mnennost finger 

46. 'n.me.hodoTJ^^4twrere^fo.:pr^^ "S'^^TmS^ '^"^^^ 

the hands are recognized by adding attracts Bo^ neTf^r i. ^ 7^ '"•^'hod of claim 56, wherem the thumb factor is 

center of the attrac^int'rC?^iS.^^^t£ 60 Jlf/ ^=7^.°°' ^ velocity of the imiennost 
the distances to the folate. atLctoT^ s^thaS^ T ""^"^J!" """^ ^ 

are assigned to the fore^ attra^r So^y J^e £ ^TU" *'™'> P'«*°«» or motion, 

pioduces enough contJ^ nearly K Sa.l7nW • ^ir °^ ^' ^^""^ ^ verification step 

47. -n^ meTod of claimXl^Jin^he^S^^Tan ff^XTilT^^T ''■'^^^ 
attractor template corresponding to parts of a pLticular hand 65 60 ^Za^^^^^^ ^ mmunizabon step. 

is translated to remain «mered^n the last estkM^«^J^^ • ^ "^J*^ Oi«*«""e s"rfa« «»ntacts and establish- 
of that hand estimated posiUon mg finger, thumb, and palm identities, the method compris- 

ing the steps of: 
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to be from a gnren hand, contacts into left hand chisters and right hand chuiteni: 

pa^g through each conUct once to form an ordenid assigning finger and pato identities torconlacts witlS 

_ . each cluster, 

SSntrrr* l"*"" '° ^^ o:6^'^<i loop; s computing for each partition an assignment fitness mea- 

deteraumng whether the innermost contact b a thumb. sure which represents the biomechiaical consistency of 

fingerup or pahn cootoct from conUct and inter- the fit of conUct dusters to their assigned attractor 

contact features of the mnermost contact; and rings; iu.ii.iut 

assigning thumb, fingprtip or pahn identities to non- choosing the parUtion which has the best assienment 

innermost contacts based upon the features of the lo fitness measure as the partition containing the true 

contacts, assignment of the mnermost contacts, vertical contact identities; and 

p^t^n relative to assigned contact^ and the loop recognizing each hand's configuration &om the combi- 

jrt A *- 17 nation of and features of surfiace contacts assicmed 

61. An warat,^ for distrngmshing pahn heel contacts within each attractor ring of the best^Sn 
from other types of hand contacts in a wstem for recoeniz- , tin tiw. "?^^""8 ™ ™ parntion. 

ing hand activity on a multi-touch surfed and g^S ' Jtl^ of clam. 65. wherem the hand assign- 
input signals to /competfag device therefromThe^^S ^t^^'^^T ^K-. 

comprising: conUcts are stabilizing. 

means for finding the nearest neighbor contact of a given f..^!; '"''-1'°^ °^ ""'T '^^'^'^ ^ reactivated path 

conUct in a pLie of the suif^; and ^ temporanly-removed contact regains its previous iden- 

means for suppressing identification of the given contact ^ ^^^^ ^..k^ r i • - 

as a palm heel contact it a neighbor contSt exists and • ^ "'^^^^ wherem each attractor point 

is closer to the given contact than the anatomical ^ ^ expected portion of a corresponding hand 

separation between inner and outer portions of a palm ™ 

heel. The method of claim 65, wherein the partition gen- 

62. The apparatus of claim 61, wherein the finding means ^ ^^ating step comprises the following sub-steps: 

ignores contacts identified as forepalm hand contacts. constmcting approximately vertical contours between 

63. An apparatus for distinguishing palm heel contacts horizontally adjacent contact; and 

frooi other types of hand contacts in a system for recogniz- constructing a partition from each contour by tenUtively 

mg hand activity on a multi-touch surface and generating assigning conUcts \^^ch are positioned to the left of a 

mput agnals to a competing device thercfrx>m, the apparatus 30 contour to the left hand cluster and contacts to the right 

compnsmg: of a contour to the right hand cluster. 

means for measuring the total proximity, orientation, and 70. The method of claim 69, wherein the hand assign- 
eccentricity of all contacts; ments of previously identified contacts can be locked so to 
means for encouraging identification of a given contact as depend on which side of the dividing contour the 
a pahn heel contact if its ratio of total proximity to contacts lie, while assignments of new contacts still depend 
eccentricity is larger than for a typical fingertip contact; °° which side of the contour they lie. 

71. The method of daim 65, wherein each attractor ring 

means for encouraging identification of a given contact as translated, scaled and/or routed to match previous posi- 

a palm heel contact as its orientation approaches the estimates for the hand coiresponding to the attractor 
expected slant of a pahn heel. ^ 

64. An apparatus for distinguishing thumb contacts fiom 7^: ^® ™ethod of claim 65, wherein the attractor points 
other types of hand contacts in a system for recognizing within eadi ring are individually ofiket by previously esti- 
hand activity on a multi-touch surface and generating input offsets. 

signals to a competing device therefrom, the apparatus ^* method of claim 65, wherein the as^gnment 
comprising: 45 fitness measure is a total cost computed as a weighted sum 

means for measuring the aze and orientation of all &om each contact to its assigned attractor point 

contacts; ^ the attractor ring of its assigned hand cluster, and wherein 

means for encouraging identification of a given contact as ^^^est total cost, 

a thumb contact if its size is larger than a typical . w I f "^^"^ '^^ wherein the distances 

fingertip contact; ^° between eac* surface conUct and each attractor point are 

means for discouraging identification ofa given contact as ^^^^^ according to how closely measured conUct 

athumbconual^sizeislargerthanfj^i^^^ ^Sci^ o " ^"""h'- ^"^^^^ 

conUct; and eccentncily, onentation, distance to nearest neighbor 

mea^forencouragingidentificationofagivencontactas ss raS«ci;;^S'S^tI""'"'^^'=^ "^'^ ''^ 

e'xXs^"?.^!.^™"''"' "^"^"^ ^75.Themethodofc,aun73,whereinaseparationbetween 

65. A method fi,r determining' which hand causes each T''^ "^^^ innermost finger is corn- 
surface contact detected on a multi-t^^r^»X^ T " "f^fi""^" ^T"" ""'""""^ '^"^^^ =^ 
input signals generated by hand activityonS:^^^' «, Sh ESSftot^^J^T^ ' ^""^T ^^""^ 
depend on the identity of the hand peiforming the activitv 1^^,^^^ . ^ ""^ 

and so that multiple hLls can perfoW^^I^nt^W- bromechanically consis- 

i?^°ps"'ofr^'"* simultaneously, the method «,mprising 76. TKe method ofdaim 73. wherein a hand portion of the 

defirnng a Ujmphtte of hand part attractor points on the « L'iioTwhll^rJge'L^tr^^^^ 

^rfo^iS^a^g;'"""^ ^ '^/^^^'^catesthehidisretuLngtoi^'^r 
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77. The method of claim 73, wherein a haod portion of Ihe 
total cost is increased by a palm cohesion weighting function 
when the contacts assigned to the palms of a hand are 
scattered over an area larger than the anatomical size of an 
outstretched palm. 

78. The method of claim 73, wherein the total cost of a 
partition is increased by a interfaand separation weighting 
when the measured separation between contacts tentatively 
assigned to opposite hand clusters indicates that the hands 
may be overlapping or close to touching. 

79. A method for integrally extracting multiple degrees of 
freedom of haod motion from sliding motions of two or 
more fingers of a hand across a multi-touch surface, one of 
the fingers preferably being the opposable thumb, the 
method comprising the steps of: 

tracking, across successive scans of the proximity sensor 
array the trajectories of individual hand parts on the 
surface; 

finding an irmermost and an outermost finger contact &om 
contacts identified as fingers on the given hand; 

computing a scaling velocity component torn a change in 
a distance between the innermost and outermost finger 
contacts; 

computing a rotational velocity component from a change 
in a vector angle between the innermost and outermost 
finger contacts; 

computing a translation weighting for each contacting 
finger; 

computing translational velocity components in two 
dimensions from a translation weighted average of the 
finger velocities tangential to surface; 

suppressively filtering components whose speeds are con- 
sistently lower than the fastest components; 

transmitting the filtered velocity components as control 
signals to an electronic or electro-mechanical device. 

80. The method of claim 79, wherein the scaling velocity 
computed from a change in distance between the innermost 
and outermost finger contacts is supplemented with a mea- 
sure of scaling velocity selective for synunetric scaling 
about a fixed point between the thumb and other fingers. 

81. The method of claim 79, wherein the rotational 
velocity computed fi^om a diange in vector angle between 
the irmermost and outermost finger contacts is supplemented 
with a measure of rotational velocity selective far symmetric 
rotational about a fixed point between the thumb and other 
fingers. 

82. The method of claim 79, wherein the trarslation 
weightings of the irmermost and outermost fingers are 
constant but the translation weightings of central fingers are 
inversely related to polar component ^>eeds so as to prevent 50 
vertical translation bias while performing hand scaling and 
rotation but otherwise-include all available fingers in the 
trarslation average. 

83. The method of claim 79, wherein the translational 
weightings are related to the ratio of each finger's speed to 
the speed of the fastest finger so that if the user chooses to 
move fewer fingers than are on the surface the gain between 
individual finger motion and cursor motion does not 
decrease. 

84. The method of claim 79, wherein the suppressive 
filtering step comprises the following two sub steps: 

dowoscaling each velocity component in proportion to a 
function of its average speed compared to the other 
average component speeds; 

dead-zone filtering cadi downscaled velocity component 
wherein the width of the dead-zone depends on the 
distribution of the current component speeds. 
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85. The method of claim 79, wherein the orientation of an 
ellipse fitted to the thumb contact after each successive 
sensor array scan is transmitted as an additional degree of 
fireedom control signal. 

86. A method for integrally extracting roll and tilt degrees 
of freedom of hand motion from pressure changes of three 
or more non-coUiiiear hand contacts comprising any of 
thumbs, fingertips or palms, the method comprising the steps 
ot 

tracking across successive proximity images the trajecto- 
ries of individual hand parts on the surface; 
measuring proximities from each hand contact in a cali- 
bration proximity image once all available hand con- 
tacts have been stabilized; 
computing an average hand contact position from a post- 
calibration proximity image, wherein all hand contacts 
are weighted equally; 
computing a weighted average hand contact position from 
a post-calibration proximity image, v^erein eadi hand 
contact is weijghted according to the ratio of its current 
proximity to its caHbrated proximity; 
computing for each post-calibration proximity image the 
difference vector between the weighted average hand 
contact position and the average hand contact position; 
dead-zone filtering the difference vector to remove varia- 
tions in proximity due to unintentional posture shifts; 
and 

transmitting the filtered difference vector from each post- 
cah*bration proximity image as roll and tilt control 
signals to an electronic or electro-mechanical device. 

87. The method of claim 86, wherein reference proximi- 
ties slowly adapt to decreases in individual contact proxim- 
ity. 

88. The method of claim 86, wherein the ratio of current 
hand contact proximity to its calibrated hand contact prox- 
imity is dipped to be greater than or equal to one. 

89. The method of claim 86, wherein an additional total 
hand proximity component is computed from the average of 
all current hand contact proximity to calibrated contact 
proximity ratios, and the total hand proximity component is 

^ transmitted to computing device, 

90. The method of claim 86, wherein the compiUation and 
transmission of hand roll and tilt rotational axes are initial- 
ized by resting all five fingers on the surface, tapping palms 
on the surface, and then resting palms on the surface. 

91. A manual input integration method for supporting 
diverse hand input activities such as resting the hands, 
typing* multiple degree-of-freedom manipulation, command 
gesturing and handwriting on a multi-touch surface, the 
method enabling users to instantaneously switch between 
the input activities by placing their hands in different con- 
figurations comprising distinguishable combinations of rela- 
tive hand contact timing,_proximity, shape, size, position, 
motion and/or identity across a succession of surface prox- 
imity images^ the method comprising the steps of: 

tracking each touching hand part across successive prox- 
imity images; 

measuring the times when each hand part touches down 

and lifts off the surface; 
detecting when hand parts toudi down or lift off simul- 
taneously; 

producing discrete key symbols when the user asynchro- 
nously t^, holds, or slides a finger on key regions 
defined on the surface; 
producing discrete mouse button click commands, key 
conmtiands, or no signals when the user synchronously 
taps two or more fingers from the same hand on the 
surface; 



30 



35 



45 



55 



60 



65 



73 



US 6323,846 Bl 



15 



producing gesture commands or multiple degree-of- 
frccdom manipulatioa signals wben the user slides two 
or more fillers across the surface; and 

sending tte produced symbols, commands and manipu- 
lation agnals as input to an electronic or an electro- 
mechanical device. 

92. The method of claim 91, wherein production of 
discrete key symbols or mouse button click commands from 
single finger isps or finger chord taps is accompanied by 
transmisaon of activation signals to a light or sound feed- 
back generating device. 

93. The method of claim 91, wherein accidental synchro- 
nous finger taps during typing arc prevented from diiipting 
the typing session by not producing input signals to a 
computing device when the accidental chord Up is the first 
detected chord Up since the last detected asynchronous key 
Up, the chord tap occurs within a typing timeout interval 
^ibscquent to the last detected asyndironous key tap, and no 
finger slides have been detected between the last detected 
asynchronous key Up and said accidental chord Up. 

94. The method of daim 91, wherein hand resting is 
tolerated by siCTressing a generation of output commands 20 
when aU^ or nearly aU of the fingers simultaneously engage 
the multi-touch surface and remain substantially stationary. 

95. The method of claim 91, wherein user handwriting 
activity is distinguished from other input activities by its 
unique pen grip hand configuration with the following ,c 
additional steps: ^ 

esublishing the finger or palm identity of each surface 
conUct; 

measuring the relative positions and proximities of the 
identified conUcts to determine whether the inner fin- 30 
gers are pinched while the outer fingers curl under the 
palm exposing their knuckles to rest on the surface; 

entering a handwriting mode for the hand if the above 
unique finger arrangement is detected; 

producing inking signals from the motions of the inner 35 
fingers on the sur&ce while in handwriting mode; 

producing styhis lift signals each time the inner fingers lift 
off the surface while in handwriting mode; 

sending the inking signals to an electronic device for 
capture, display, or recognition; and 

leaving the handwriting mode after the hand has lifted and 
remained off the surface for a substantial time or if a 
non-pinched finger configuration is measured. 

96. The method of claim 95, wherein while in handwriting 
mode but the inner fingers are lifted, hiding and Upping 45 
motions of the palm heels produce cursor manipulation and 
clicking signals whidi are sent to the electronic device. 

97. The method of claim 95, wherein a styhis held 
between the pinched fingers touches the surface instead of 
the pinched fingers themselves to indicate pinch 
configuration, and wherein inking signals are measured from 
motion of the stylus. 

98. The method of claim 91, wherein the layout of key 
regions defined on the surface is morphed to fit the user's 
hand size and current position, the method comprising the 
following steps: 

definii^ a default key layout whose home row key regions 
lie roughly at predetermined default positions of the 
fingertips; 

identifying what hand part each surface conUct comes 
from; 

detecting a layout homii^ gesture when all five fingeis of 
a hand arc placed on the surface in a partially closed 
posture; 

measuring during the layout homii^ gesture the position 
oflfeets of the homing hand and fingers with respect to 
the predetermined default finger positions; 
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translating the key regions normally typed by the hand by 
the measured hand and finger oflfeets sudi that each 
home row key region lies at approximately the mea- 
sured position of its corresponding finger; and 
5 updating the displayed portions of the key region sym- 
bols on a visual display embedded in the surface. 
99. The method of claim 91, wherein the layout of key 
regions defined on the surface is morphed to fit the user's 
hand aze and current position, the method comprising the 
10 following steps: 

identifying what hand part eadi surface contact comes 
from; 

detecting a layout homing gesture when all five fingers of 
a hand are placed on the surface in a partially closed 
posture; 

measuring during the layout homing gesture the position 

of each finger on the surface; 
translating each home row key region and its neighboring 
keys by an amount such that the new position of the 
home row key region is approximately the same as the 
measured position of its correspondi^ finger; and 
updating the displayed portions of the key region sym- 
bols on a visual display embedded in the surface. 

100. The method of daim 91, wherein typing while the 
fingers mostly rest on the surface is made easier by not 
requiring finger liftoff quiddy foUowing each press of a key 
region, the method comprising the following steps: 

measuring the relative impulsiveness or forcefulness of 
30 finger touchdowns; 

producing key symbols from liftoff and impulsive or 
forceful touchdown of a finger while most fingers on 
the same hand are resting on the surface even if the 
finger continues to rest on the surface without quickly 
lifting bade off the surface; and 
not producing key symbols when finger touchdowns are 
gentle or synchronous with other fingers. 

101. The method of claim 91, wherein typematic or 
automatic key repetition wb&n a finger is held on a key is 
emulated de^ite the fact that fingers which sUy on the 
surface for extended periods are normally ignored to support 
hand resting, the method comprising the steps of: 

issuing a first keypress signal after a holding finger has 
touched down and remained on a deared key region for 
at least a hold setup time interval and all other fingers 
on same hand leave the surface within a release setup 
time after holding finger touched down; 
periodically issuing additional keypress signals every 
repeat time interval subsequent to the second keypress 
signal as long as the holding finger continues touching 
the desired key region; and - 
ceasing repetitive issuance of the additional keypress 
signals ^en the holding finger lifts off the surface. 

102. The method of claim 101, wherein touchdown, 
resting or liftoff of hand conUcts identified as pahns on 
either hand does not affect the typematic sUte. 

103. The method of claim 101, wherein the cycle of 
keypress signal generation continues irrespective of whether 
other fingers touch down and rest on the surface subsequent 
to issuing the first keypress signal. 

104. The method of daim 101, wherein the repeat time 
mterval is continuously adjusted to be inversely proportional 
to current measurements of holding finger proximity or 
pres^ire. 

105. A method for choosing what kinds of input signals 
wiU be generated and sent to an electronic or electro- 
medianical device in response to Upping or sliding of 
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forming a subset any two or more'fingeis which touch 

bmation of finger identities in thTS^f" 
generalmg input signals of this new kind when further 
distochve motions of the fingcts o^ 

kinds of mput signals in response to liftoff and 
clHonous touchdowns untiTTfingets MoS ^ ^ 

whlJ^-^'n^r l^JTi^/r ^.^^^^^ 

same kinds of ^uTS^n^^. u i^^'^ the 
input signals ^ ^ ^ ^^ds of 

dently fiom contacts iS«l^ rii.^^^^/"^ ^WmovemTntTe'l^'tv'T. 1^? ^^"^ 

109. TTh> method of cSS^M^^Ko'^r-. • . °'«=»li»g eSeS^f^^^'*.'''^?"'^*'^ 

UO. Tl.e method of claim 105 wK • ^ A i^'^fo^lT' ^ a« li^d- 

distinctive finger motiolt^i^Li^JSS J""! n,ulti-toi:h^J'^ -'W^ «e^s performed on a 
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device o^^ , ^ •^8'=""^ """"o" of a touch 

torde":s.^iiSr^rrsS- rs'*. 

.otion^r^S^L^S^^^ 
the method comprising the following st^- 
mcasmng. storing and transmitting to a computing device 
^ or more representative tangential vel^Sf £Z 
touch device man^ulation; "'umg 

computing and storing a liftoff velocity from touch device 
posuions mimediately prior to the touch devi« S 
companng the liftoff velocity with ih, ~- 

ously movmg the cursor if a tangential liftoff dii^nn 
Wmately e<^ Uie ^rcscni.^ ^^ 

LSS'S^""'^ --^'^^'^ Of representative 

""SS to^'^'"^^ """^ -^aJs after 
Ufloff to a oompulmg device such that Te ausor 
movetnen, velocity «,rre^^ ^ one of Ltp^. 
lahve tangential velocities; and rcpresen- 

*Sn Signal- 
When the touch devK« engages the surface aaZ 

ovc no tarther or a window can scroll no 

n^P^nJ^e'-^^nti^^lSieVJa^^X"™ °^ 
several instanta,«5us vel^^ " "^'^hted average of 

is S-tou^h^^t ^^^ri'"^ ^ 

cu-or movemem^wt Tt^.?^ ^'-- 'b^ 



- ulc select 

new subset of oni^ fin^^^^ ^ ^ ^ nngcr forms a 
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. "*4/**auig uic Steps ol: 

generating a cut to clipboard signal in resnonse .n 
Pinchmg motion between the t^blTS^rtiV 
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